2021 · Semester project, ETH Zürich
GPU-accelerated Bayesian inference
Moving the dense linear algebra behind approximate Bayesian inference onto the GPU, and measuring where that actually pays off.
- CUDA
- cuSOLVER
- C++
- R
- nvprof
- LU factorisation ported to the GPU and profiled against the CPU path
- Transfers overlapped with compute so the PCIe round trip stopped dominating
- Crossover point measured — below a problem size the CPU still wins
R-INLA estimates latent Gaussian models using the integrated nested Laplace approximation. It is the pragmatic alternative to MCMC: instead of sampling the posterior, it approximates it directly, which for a large class of models is both far faster and accurate enough.
The cost sits in repeated matrix factorisations. That is precisely the shape of problem a GPU should win.
Why it is not automatic
A GPU is fast at arithmetic and slow to reach. Every factorisation means moving a matrix across the PCIe bus, computing, and moving the result back. For a small matrix the transfer dominates so completely that the GPU version is slower than the CPU one — you have paid the latency and barely used the arithmetic.
So the project was not "port the factorisation to CUDA". It was: find where the crossover sits, and get the transfer out of the critical path.
What that meant in practice
Factorisation on the GPU via cuSOLVER, replacing the CPU LU decomposition in the inner loop.
Overlapping transfer with compute. CUDA streams let you copy the next matrix while factorising the current one. Done properly the transfer cost largely disappears behind work that was happening anyway. Done carelessly you serialise and gain nothing — the API does not stop you.
Profiling rather than guessing. nvprof tells you what the kernels
actually did: occupancy, memory throughput, whether you were compute-bound or
waiting on a copy. Almost every intuition I had about where the time went was
wrong at least once, which is the usual experience and the reason to measure.
The result that mattered
Not a single speedup number — a curve. Below a problem size, the CPU wins and the GPU implementation is a liability. Above it, the GPU pulls away and keeps pulling.
Reporting the crossover rather than the best case is the honest form of the result, and it is the form a practitioner can act on.
What I took from it
"Accelerate it on the GPU" is a hypothesis, not a plan. The arithmetic is the easy part; data movement decides the outcome.
Overlap is where the real gains are, and it is invisible in the code — you only see it in a profiler timeline.
A performance result without its conditions is marketing. I still write benchmarks this way: here is the curve, here is where it crosses, here is the hardware.
Four years later I hit exactly this again — a 7B language model taking ten minutes per completion because layers had silently spilled to the CPU and every token was crossing the bus. Different decade, same lesson.