DPM-Solver++: Making Diffusion Sampling Fast
A trained diffusion model still has to be sampled from, and the naive way (reverse the forward noising process one small step at a time) takes hundreds to a thousand network evaluations per sample. That’s the practical bottleneck DPM-Solver++ is built to remove.
Sampling is solving a differential equation
The reverse diffusion process can be written as an ODE describing how a noisy sample evolves back toward data as noise decreases. The naive DDPM sampler solves that ODE with the simplest possible numerical method, tiny first-order steps, which is accurate but requires a huge number of steps to avoid error. DPM-Solver++ instead treats it like what it is: an ODE with known structure, solvable with a proper higher-order numerical solver that takes much bigger, still-accurate steps. That’s the whole idea: fewer, smarter steps instead of many naive ones.
Why “++” over the original DPM-Solver
The original DPM-Solver already sped things up substantially, but it becomes unstable with classifier-free guidance at higher guidance scales, exactly the setting most practical diffusion models actually use to control output quality. DPM-Solver++ reformulates the solver around predicting the data directly (rather than the noise) and adds thresholding, which keeps it stable in the guided regime instead of diverging.
What it looked like in practice
In my thesis work on diffusion models generating text non-autoregressively in BERT embedding space, sampler choice was as consequential as architecture choice. With DPM-Solver++, I reduced discretization error to 0.65 while cutting the number of sampling steps far below what a naive first-order sampler would need for comparable quality. A model that’s excellent on paper but takes a thousand steps to sample from isn’t usable in anything latency-sensitive. The sampler is what turns “produces good outputs eventually” into “produces good outputs fast enough to matter.”
The takeaway
It’s tempting to think of the sampler as a minor implementation detail sitting downstream of the “real” modeling work. In practice it’s the difference between a diffusion model that’s a research curiosity and one that’s fast enough to actually deploy. Treating sampling as ODE-solving, not just noise-reversal, is what gets you there.