As I'm sure many of you noticed, November was a turbulent time for nopy. Not only did we hit a new record with 9.3 petabytes transferred, in mid November a series of very large files were uploaded which also proved incredibly popular. This resulted in almost triple our normal traffic, although the big issue was the sheer size of the files, slowing nopy to a crawl and even crashing a few servers in both the EU and NA cache cluster.
The Problem
The way the cache cluster works is to offload very popular files to a dedicated cluster, thereby improving speeds for popular files whilst also reducing load on the normal file servers. The cache cluster featured servers with 1 or 2 SSDs and 32GB RAM, and until now this setup worked reasonably well, whilst we did see some occasional issues hitting the memory limit, these were rare and temporary. Additionally, many of the SSDs on these servers were older models, due to our budget limits.
Unfortunately the perfect storm in mid November completely overwhelmed these servers. We very quickly hit the memory limit, and the legacy SSD storage couldn't keep up with demand, resulting in incomplete transfers for users. Servers with 2 SSDs favoured much better, but still eventually had issues with lack of RAM and legacy SSD speeds.
The Solution
Thankfully, over "Black Friday" we have been able to secure some good deals with a new provider to replace the entire EU and NA cache clusters. All servers in both clusters now have double the RAM (64GB) and 2x Samsung 850/860 Pro SSDs. Additionally, we've been able to secure 10Gbps dedicated ports for these 10 new servers, bringing our total theoretical network capacity to well above 100Gbps. We're also in talks with another provider about the US file cluster, which we'll hopefully have news about soon.
As a non profit service with a finite amount of resources, it's unfortunately inevitable that there will be turbulent periods where our capacity is maxed. Thankfully due to the incredibly generous support of our community, we're able to stay ahead of most of these issues, and upgrade before any real problems hit. Sadly that wasn't the case this time, and we suffered severe impacts to our service. However, the changes and upgrades we've made over the past week should put nopy in a much more stable position to weather the next storm
As always, thank you for your support, without which nopy wouldn't exist <3