This is the second of two updates, concerning the recent WordPress move that most people probably have not noticed.

What people probably have noticed is that the WordPress server stability has been less than great recently. The reason for this is, essentially, unseen screwups in the original setup that I wasn’t looking out for because of AWS credits that finally expired.

Basically, what used to cost near $0 suddenly became $150 per month halfway through. The reasons for this are:

1. Overprovisioned EC2 server, just in case.
2. Separate RDS server, in a different Availability Zone
   • I used RDS because I didn’t want to have to handle backups, but in practice, the UpdraftPlus extension did its job admirably, making it irrelevant.
3. Orphaned EBS volumes built up over time when I last refactored the servers

I adjusted #1 down, which mostly worked most of the time. When it didn’t, EC2 choked so badly it couldn’t even force-stop the instance without timing out, delaying any recovery.

I tried to change #2 by creating a multi-AZ mirror and failover, but it still ended up in a different AZ from the EC2 instance and the entire server was down while I was trying this.

I was able to fix #3 without much incident by just carefully auditing the servers. I encountered a bizarre bug where two different EBS disks had the same UUID so I had to learn that I needed to override the check for duplicate UUIDs to inspect the disks for their contents and backup as necessary.

Throughout all this, the server logging was also not up to snuff as the WordPress docker image was pretty simple to get running but running a server in a Docker network in a VPC made configuring the proxy protocol too much of a headache, hence the server was never really accounting for IP addresses properly. After looking at the TCO, I figured that switching to a different VPS was going to be worthwhile - better $150/year than optimizing down to, at best, a multiple of that while wrangling with AWS services. It’s less elastic, but we don’t really need elasticity for this.

So, I set MariaDB, nginx, PHP, letsencrypt, and WordPress up (bonus points: I got to migrate off of a grossly out-of-date traefik and a letsencrypt cron docker image which didn’t run periodically as it advertised), and went to town on moving it over. The result is a mostly transparent move, though there may be some hiccups (tag me [Yewnyx] in the Parahumans Discord to let me know).

Also, I discovered that the tiered caching I put in place (first CloudFlare, then the WP extensions) was saving substantial CPU. I discovered this in part because now that the logs aren’t all messed up, I was able to notice a scraper scraping the site every couple of seconds. Wildbow writes fast, but not THAT fast. Whoever is doing that, kindly slow the fuck down, please!

Anyways, fingers crossed things should be more stable now - and a great deal cheaper for me, too.

P.S. Shoutout to etckeeper for helping me feel more confident in recording the setup process so I can repeat it later. Orchestration and containerization are cool but sometimes you just want to set up a server.