I had my server hosting this webpage get pinned on memory usage today. I think the problem was just that my server only has 1G of memory, and running docker + apt daily upgrade + fwupd all at once caused it to get stuck.

I did not reach OOM logs in the syslog though i only ever got Under memory pressure, flushing caches which is probably still enough to cause the unresponsiveness I saw when trying to ssh.

Ultimately, I just added a swapfile to at least give me some breathing room next time this happens. Until then, ill be trying to stress test the system to get an idea of what types of things will break the server.

first round was wrk:

wrk -t12 -c400 -d30s https://nikopoulos.xyz/
Running 30s test @ https://nikopoulos.xyz/
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   571.71ms  760.04ms   1.92s    68.53%
    Req/Sec     9.11      9.90   101.00     84.42%
  1209 requests in 30.07s, 9.66MB read
  Socket errors: connect 1570, read 542, write 0, timeout 923
  Non-2xx or 3xx responses: 184
Requests/sec:     40.20
Transfer/sec:    328.93KB

It was pretty cool to see uvicorn pop-up on top due to high CPU usage, but uvicorn's memory usage never exceeded 35% (with 29% as a rough baseline)

This is my rat-dog. Tobi Satan. Despite his charming appearance he is solely responsible for my rapidly deteriorating mental state. If you see him in the wild, do not approach. Tobi will use any manner of trickery to separate you and your hard earned cash.