blog: Tongfamily.com down out of disk so use Block Volumes after failing with DigitalOcean Spaces and Ingix

OK, I’ve been running Tongfamily.com on Digital Ocean for the last five years and it has been fairly trouble-free (particularly when I added the 2GB swapfile) to prevent out-of-memory problems, but the last two days it has been crashing. It turns out that my simple installation with a 25GB disk has been running out of memory because of all the images that I’ve been using in posts.

So a few things that I fixed because of deferred maintenance (eg me not paying attention):

There is a new web-based console that requires installing some Ubuntu packages. The problem is that the old web console was disabled, so you have to have SSH access to the machine or use their recovery console to install it. It is one of those pipes from some random URL into bash which is a little scary, but it does work.
When I looked into this and i saw that /var/log was taking up 10GB, I saw a journal file was taking up over 10% of the disk space, but a quick journalctl --vacuum-size=1G command solved that and I capped the journaling at 1GB this game to breathing room. The other big file was in auth.log which is the authorization but most of it was 300MB in the main log, so I left it.
The problem was in /var/www/html which is where WordPress installs itself, let me say that du -sh for summarizing the disk size in human-readable format if very useful as is df -s to look at which volumes are full. And I found that the WordPress installation was nearly 15GB with most of it being in /var/www/html/uploads which is where the images for the blog post tops go.
The problem is that Digital Ocean only lets you upgrade the size of the droplet one way, but really storage should be separate for the asset files. You do want to leave the main installation on the droplet for speed. There are a few choices here, you can use Digital Ocean Block Volume storage for $2.50/month or you can use the new Digital Ocean S3 compatible bucket storage called Digital Ocean Spaces. The nice thing is that 80% of my space is taken up with image uploads, so either would work, but the S3 does have a CDN built in and sets me up for static hosting someday, so I thought I would try that.
This requires that you create a new one Digital Ocean Spaces of these, get the Access Token and the Secret Key from the API, and then you can use s3cmd to access it or you can use the Sync to Spaces WordPress plugin. Then you enter that information into Spaces Sync.
You can use s3cmd to access any S3-compatible storage system which is pretty nice, you just run a config command and you can connect directly to the S3.
Note that I did try to use a third-party system that is S3 independent and even set up their image processing cloud pieces. But when I turn it on, everything basically stopped. So I uninstalled that.
This seems to only work for new Images and videos that are uploaded, but this post is how I’m going to try it. The main problem was that I got the images to upload including the thumbnails, but they would not display properly, but it seems to work fine if you do dual storage (which kind of defeats the purpose.

Digital Ocean Spaces deprecated and Using Blocks

The first thing to notice is that buried in the documentation is that there are two modes that you can run, the Spaces can have the images, or both the server and the Spaces can. One big problem is that the plugin is now deprecated, so it isn’t clear how to do this in the long term, so I just installed the Block Device for now

The DigitalOcean Spaces Sync first needs the Connection Settings which are the Spaces Key, the Secret Token, then the container name, and finally the endpoint. So for instance, if the Spaces location is https://foo.nyc1.digitaloceanspace.com, the foo is the Spaces Container (confusing, this is not a docker container but just an S3 bucket equivalent) and the endpoint is nyc1.digitaloceanspaces.com
The confusing this is what the Full URL-path to files is. Basically, if you use something like a full URL path like http://tongfamily.com/wp-content/uploads then it is mirrored, but if you use a storage public domain then it is only stored in cloud storage. Confused yet? I’m confused. It needs something called the storage public domain I’m very unclear on what this means, but it looks like you can somehow set a DNS CNAME to some location.
The solution appears to be that I can get the mirrored thing to work by putting in the wp-content/uploads URL above, but can’t figure out how not to use the local server.
So I just created and a Block device and then edited /etc/fstab with the right entry to first create a uploads.new, then copy all the uploads over and then finally move that and then remount the fstab at uploads and everything is good. The major commands are to first create a mount point with sudo mkdir /var/www/html/wp-content/uploads.new, looking the /dev/disk/by-id mount point since you should not use /dev/sdb for instance since these points can change on boot then editing /etc/fstab and adding the line sudo mount -o defaults,nofail,discard,noatime /dev/disk/by-id/scsi-example /var/www/html/wp-content/uploads.new.At this point, you have the original uploads, so it is easy to rsync -av /var/www/html/wp-content/uploads /var/www/html/wp-content/uploads.new which will copy the files to the mount device, then you can mv uploads uploads.new and edit the fstab to use the regular uploads. Here the findmnt --verify --verbose is very useful as it parses the /etc/fstab. And if it works then mount -a reads the table and mounts it.
This method works, but when creating a new post, the deprecated plugin doesn’t work, so time to just remove the whole thing and delete it

Trying and Failing with Imgix

OK, before I settled on the Digital Ocean Space Sync plugin, I tried a third party one called Ingix, which seemed to work but unfortunately hung the WordPress installation. It turns out that the free version doesn’t allow migration of images, you have to pay for that, so I ended up uninstalling it for both those reasons, but here is how to install if you want to try yourself:

Installing the Ingix and having it secure means adding a security access private key so that there is a digital signature when you download the images
The installation of the sync was pretty easy, but man the options were really daunting, the Digital Ocean thing in comparison is much simpler, but Ingix does allow image hosting for up to 1,000 original images, so I might look at that some day.