Gaussian splats part 3: There’s always a bigger scene — Magnopus
Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,307 words · 16 segments analyzed
Previously, on OKO…The year was 2025, and as I wrote last time, the industry had fallen head over heels for Gaussian splats, and Magnopus right along with it. I'd spent a good while dragging our Unreal splat renderer out of Niagara and rebuilding it around the engine's instanced rendering: every splat registered as an instance, a GPU-side sort on each frame was invoked to composite the transparency in the right order, and a material shaded each one into a soft ellipse.It worked, and it worked well enough that I wrote a whole article about it. I felt rather pleased with myself.Well, as it turns out, I celebrated way too soon when I claimed a capability of rendering unlimited splats. In reality, several walls were making this categorically untrue, and the first one was much closer than I expected.Not so unlimited after all…The first wallHere's a humbling thing I only learned after publishing that last article: the renderer I'd been showing off had a hard cap of about 16 million splats per asset.
I hadn't the faintest idea.
Every asset I'd tested happened to sit under it, but the limitation was always there.
In my hubris, I just didn’t know about it yet.That limit came down to where the per-splat data actually lived. Registering every splat as a modern mesh instance in Unreal meant handing it to the engine's GPU Scene, effectively a central ledger of typically everything in the world that resides on the GPU.And while GPU Scene is superb at what it does, it was built to support thousands of instances of a mesh – not tens of millions or hundreds of millions – under the hood is a cap of at most 16 million instances of a given mesh. When you cross it, things come apart; only part of the asset is visible at any given time, graphical corruptions abound, and you get a warning printed on the viewport informing you of your failure. The naive solution, and the one that worked out for me, was to stop using the GPU Scene to track splats altogether. Instead, I went fully old-school and relied on a more traditional instance-render pipeline that more or less ended up being a direct invocation of what’s generally known as DrawIndexedInstancedIndirect. This required a few key steps:Skip registering instances with the GPU scene; do a classic indirect instanced draw instead.Feed the indirect draw buffer the draw count from the sort keygen shader, which was already iterating over all splats while generating sort keys per-frame.Store all splat instance data on the GPU via textures (which we also got for free from the last article).Write a small vertex factory (Unreal's abstraction over vertex shaders and their material bindings) and have the vertex shader sample the sorted splat buffers (see the last article), get the appropriate splat attributes and then pass that onto the material.The rendered image came out pixel-for-pixel identical to before, but the 16-million ceiling evaporated, because the thing imposing it was no longer in the loop. Our test asset of ~30 million splats was rendering happily. Better!Free from the shackles of GPU Scene’s max instance count.Some other nice things fell out of the custom vertex factory approach too. Since splat meshes are always trivial quads, they’re easy to derive procedurally in the shader itself, which meant there was no longer a need to specify an actual Unreal mesh – the system just derives the vertex attributes in the vertex shader, making the whole thing a little more turnkey.And the other benefits that indirect-instanced-draw effectively moved us to more of a bindless-rendering approach. Since we were already iterating over all splats each frame while generating their sort keys, we could also do more advanced per-splat culling in there, and then write the sum total of all visible splats to the indirect draw buffer. We now only drew exactly the number of splats visible within the camera frustum, and the culling logic lived entirely on the GPU. Before – GPU Scene implementation After – indirect instanced draws and per-splat culling While that’s great, in that it gives us a nice performance boost, what’s also worth calling out is that the number of splats being drawn per frame was now truly variable, and decided entirely on the GPU.
The pipeline had grown into the right kind of shape a level-of-detail system needs; drawing a different subset every frame, decided at the last possible moment.I hadn’t set out to build an on-ramp to LODs.
But that’s what it turned out to be.Hubris: ReduxLifting that instance cap felt like the finish line.
However, in less than a week I hit the next wall.Because the GPU Scene exfiltration had changed how many splats I could draw, but not how many I could hold: every splat in the scene still lived in GPU memory, all the time.
That's completely fine when the whole scene fits. But obviously a splat isn't free in terms of memory.
Position, scale, orientation, opacity, spherical harmonic coefficients, all told about 32 bytes per-splat once I'd packed it as tightly as I dared.Which implies you can throw a lot of splats at VRAM, until you meet a real photogrammetry capture of a real site, at which point you discover that reality is annoyingly high-resolution and doesn’t care about your VRAM budget.The captures we actually wanted to show people were bigger than the capacity of the machines, and there was no more clever packing to be done.
So the whole scene can't be resident at once. Which means only part of it can be. Which means the renderer has to decide, every frame, which part to render, go and fetch it and then draw it on-screen.
Stop me if you’ve heard this before.What LOD even means for splatsIf you've done anything in real-time graphics, you know about LODs: far-away things get a cheaper stand-in. A distant tree mesh becomes a lower-poly tree, then a billboard, then a smudge. Things get cheaper over distance.Splats don't have polygons to throw away, but they have something better: there are simply too many of them, and a lot of that density is wasted on a region that's forty metres away and six pixels wide. So a splat LOD is a decimated copy of a region; the same blob-cloud, with most of the blobs removed, and the survivors enlarged to cover the gaps.
At LOD level 0, we show every splat, and at higher LOD levels we show progressively fewer.The format we settled on, PlayCanvas's Streamed SOG, bakes this for you.
It essentially carves the scene into a spatial tree of regions ("leaves"), and each leaf describes several decimated LOD levels. A little lod-meta.json at the root then describes the overall tree structure and where each chunk lives. LOD Level 0 – dense LOD Level 3 – coarse Deciding what to drawSo. We have our LOD data. Now, for every frame, for every visible region, pick a LOD level.
Great, how exactly?My first instinct was to reach for something clever. My second, better instinct was to remember that the only thing that matters is how wrong it looks on screen, and "how wrong" is measured in pixels.For this, there is a decades-old idea called screen-space error (SSE), where you work out how much visual error a given level would introduce, project that into pixels at the current camera distance, and pick the coarsest level whose error stays under some threshold.
Call the threshold τ (tau), because everything in graphics eventually gets a Greek letter.For splats, we don't have a mesh to measure error against, but we can cheat somewhat with density: from how many splats a region packs into its volume you get an approximate spacing between them, project that to a pixel size, and compare against τ. Big on screen and dense? You've earned level 0.
Small and far?