Short entries, thought out loud.

I was convinced our video pipeline was GPU-bound. It’s the obvious story — large model, large frames, GPU in the middle of everything — and I was starting to think about what a faster card would cost.

Then I opened Nsight Systems, and the timeline said otherwise. The GPU was sitting idle for a good deal of the wall clock. The time was going somewhere else entirely:

• CPU-side work. Decode and preprocessing that never touched the device at all, and dominated more than I’d have guessed.
• Host-to-device round trips. Every stage that pulled a frame back to the CPU and pushed it forward again paid the transfer twice. Once you count the crossings per frame rather than assuming them, the number is embarrassing.
• Synchronisation. Copies and syncs serialise against each other, so threads spend their time waiting on a lock instead of doing work.

Concurrency made it worse in a way I hadn’t expected. Multiple processes sharing one GPU don’t actually run at the same time by default — the driver time-slices between their contexts in slices on the order of a millisecond or two, and every switch costs tens of microseconds and flushes L1. From outside, that is indistinguishable from “the GPU is too slow.”

Fixing what the profiler actually pointed at — rather than what I’d assumed — got us close to four times the throughput on the same GPU, with the same input video. No new hardware. No smaller model. The capacity was already there; it was being spent on waiting.

Which is the actual lesson, and it’s narrower than profile before you optimise: a saturated GPU and an idle GPU produce identical symptoms. Both of them look like everything is slow. You cannot tell them apart from throughput, and no amount of reasoning about your architecture will settle it. The timeline view will, in about a minute, because the gaps are the answer.

I was one purchase order away from fixing the wrong machine.