A server process kept stuttering. Short freezes where nothing visible happened, and every now and then a pause of several seconds. It runs a lot of add-on code on a single thread, so everyone’s first guess was that some add-on was misbehaving. That guess was wrong.
Measure the thread that freezes
#When work happens on one thread, the only measurement that matters is what that thread does, frame by frame. The built-in profiling options were too heavy to leave on, so I used a profiling build that records how long each phase of every frame took, which callbacks ran, and one extra number. How many major page faults the process took during that frame.
That last number turned out to be the whole story.
What the traces showed
#Almost every freeze looked the same. A long frame, a handful of major page faults, and barely any time spent in add-on code. One freeze sat inside a lookup that normally takes microseconds. That is what it looks like when data has to come back from swap before the code can read it.
Longer pauses told the same story. Sampling the stuck thread through /proc showed it in uninterruptible sleep (D state), waiting on memory, with the program counter inside the server’s own binary. Its code had been paged out and was being read back from disk.
The profiler had a bug, too
#The same pause showed two different durations in two reports. Per-frame records stored durations as 32-bit nanoseconds, which wrap at about 4.29 seconds. The aggregates used 64 bits and were right. You only find that kind of bug by comparing two numbers that should agree.
What I’d check next time
#Counting major page faults per frame would have saved me a lot of guessing. A stall full of page faults with no time spent in your code is not a code problem. If a thread is stuck in D state inside its own binary, it’s paging, and the next step is to look at what else is running on the same host and when.