Still means still

The most important thing about an image in motion is that it does not move when nothing is moving. No shimmer on the grass, no crawling shadows, no noise in the clouds. A technique that looks right in a screenshot and falls apart the moment the camera turns is no use to me, however many acronyms it has. This entry is about how the Meadowbound Engine got there on the ground, how far it is in the sky, and why several things that were supposed to help were taken out again.

A number for flicker

Flicker is easy to see and hard to argue about. One person sees shimmer, another sees wind. So it gets a number.

The camera is held still for sixty frames, after twenty to warm up, with the wind blowing. For every pixel the brightness of three frames in a row is compared: not the change from one frame to the next, but the change of the change, the second difference. A blade of grass swaying smoothly changes its brightness smoothly, and its second difference stays small. A cloud drifting across the sky does the same. A pixel that jumps up and down from frame to frame, which is what shimmer is, has a large one. The mean over the whole image is the flicker. The share of pixels that jump by more than four brightness levels, out of 255, is the second number: it says whether the noise is everywhere or in a few places.

Sharpness gets a reference too. The same view is rendered at twice the width and twice the height, with no temporal tricks at all, and averaged down. That is slow and correct, and every faster image is compared against it. A flicker number alone would reward blur; a blurred image does not shimmer, and it is also wrong.

What was wrong on the ground

Games smooth their images over time. Each frame the camera is nudged by less than a pixel, a different amount every frame, and the new frame is blended with the history of the previous ones; over a few frames every pixel has been sampled in several places, and the edges come out smooth. This is temporal anti-aliasing, and it is also where most flicker comes from when it goes wrong. Five things were wrong.

Animated surfaces had no history at all. Foliage, water and anything the wind moves were marked as “do not reuse”, because their previous position is not simply where the camera says it was. So 68 per cent of a forest frame was the raw, nudged sample, new every frame. They now reuse their history too, with a more tolerant depth test and a tighter limit on how far the old colour may differ from the new.

Every pixel stores, beside its colour, a key that says what it is and how far away. Still ground keeps its distance as a logarithm, the sky is 40, and anything the wind moves now gets the same logarithm too, turned negative and shifted below −2, instead of the old “do not reuse”:

// History key: log2(1+d) for static surfaces, 40 for sky, and -2-log2(1+d) for
// animated surfaces (water, foliage, wind-driven props). Animated surfaces keep
// temporal accumulation (depth-reprojected, neighbourhood-clamped); only -1
// (application reactive emitters/markers) has no history at all.
float taa_surface_key(bool planet, bool water, float surfaceClass, float distance) {
    if (!isfinite(surfaceClass)) return -1.0f;
    const bool animated = water || surfaceClass < 0.0f || (surfaceClass >= 2.0f && !ps_is_material(surfaceClass));
    if (!planet && !water) return 40.0f;
    const float logDistance = log2(1.0f + distance);
    return animated ? -2.0f - logDistance : logDistance;
}

The sign is the fix. A history sample with a negative key belongs to something that moves, and it is accepted if its distance lies within 0.15 of the expected one on that logarithmic scale, about ten per cent, instead of the hundredth or so that still ground has to match. A leaf that has swayed a few centimetres is still the same leaf.

Edges were always thrown out. The sub-pixel nudge makes an edge pixel alternate between the leaf in front and the ground behind it. Its history then looked like it belonged to a different surface, and was rejected, every frame, at every edge. Now history is accepted if its depth lies anywhere within the depths of the pixel’s three-by-three neighbourhood; only a real disocclusion, where the whole neighbourhood is somewhere else, still throws it away.

The image blurred itself. The history was looked up with the previous frame’s nudged camera, although the stored image had the nudge taken out. Every frame each pixel was resampled by the difference between two nudges, which is a small blur, and small blurs add up. The lookup now uses the pixel’s centre and removes the previous nudge exactly.

The blend never settled. Every pixel blended its history at the same fixed rate. Now every pixel counts how many samples it has accepted: a still pixel converges towards an average of up to 32, an animated one to 12, and both fall back quickly when things move.

The dither was grain. A tiny noise added against banding in dark gradients changed every frame, and was added before the conversion to the screen’s brightness curve, which amplified it up to about thirteen times in the shadows. It is now fixed in place, and added after.

One more thing was tried and not kept: a wider, softer filter for the current frame. It brought flicker down further, to 0.57 in the forest, but the image got measurably softer than the reference. Less shimmer bought with less detail is not the trade I want.

What it did

Measured on my M3 Pro at 1440 × 900, over the planet Elyra:

And the forest got closer to the reference, not further: its error against the supersampled image fell from 4.53 to 1.81. The frame time did not change. That was a separate problem, and it still is.

How a cloud is drawn

The clouds in the engine are not pictures on a sky dome. They are volumes: for every point in the air there is a density, how much cloud is there. To draw one pixel of sky, a ray goes out from the camera through that pixel and walks through the air in steps. At every step it reads the density. Where there is cloud, some sunlight is scattered from that point towards the camera, and some of the light from further back is swallowed.

How much light gets through a stretch of cloud follows the Beer–Lambert law: the transmittance, the share that gets through, falls exponentially with the amount of cloud on the way, T = e^(−∫σ ds). Twice the cloud does not halve the light; it squares the share. That is why the edge of a cumulus is bright and its heart is grey.

But how much sunlight reaches a point inside the cloud in the first place? That needs a second walk, from the point towards the sun, adding up the cloud in between. So every step of the camera ray starts a few steps of its own. In the engine they are four, each 2.4 times longer than the last, from 70 metres to almost a kilometre:

// Sun light march: cone of growing steps plus one long tail sample.
float od = 0.0f, ls = 70.0f, hf;
float3 lp = rel;
for (int j = 0; j < 4; ++j) {
    lp += sun * ls;
    const float3 wp = U.camPos.xyz + lp;
    const float wr = length(wp);
    od += cloud_sigma(lp, wr - R, weather4(weatherTex, wp / wr), ls, false, U, shapeTex, detailTex, hf) * ls;
    ls *= 2.4f;
}

Short steps near the point, where the shape of the cloud decides whether it is lit at all, and long ones further out, where only how much cloud there is still matters. Together they reach about 1.6 kilometres. After them comes one more sample, five kilometres towards the sun, for a storm tower that might stand between; it is bounded, so that one distant sample can darken a cloud but never black it out.

The amount of cloud on the way, od, then decides the light, and not once but four times. Light that has bounced around inside a cloud many times gets out of it more easily than a single bounce would, which is why a thick cloud is grey and not black. The usual approximation, first described for the clouds of the film Oz the Great and Powerful, adds a few fainter copies of the light that see less of the cloud and scatter more evenly:

float3 lightS = 0.0f;
float ea = 1.0f, eb = 1.0f;
for (int k = 0; k < 4; ++k) {
    lightS += ea * exp(-od * eb) * mix(phase, 1.0f / (4.0f * kPi), float(k) * 0.33f);
    ea *= 0.55f;
    eb *= 0.4f;
}

That is the expensive part, a walk inside every step of a walk, and nearly everything clever about volumetric clouds is about doing it less often without anyone seeing.

The number of steps is the whole dilemma. Too few, and thin parts of the cloud fall between them; as the cloud drifts in the wind, they pop in and out, and the sky flickers in stripes. Enough steps, everywhere, every frame, for every pixel, costs more than the frame has.

Two papers, 333 pages

Before writing more cloud code I read the two best public accounts of how it is done: Andrew Schneider’s Nubis³, 220 pages on the clouds of Guerrilla’s Horizon games, from 2023, and Alexander Müller’s Smolder, 113 pages on the volumetric rendering of 007 First Light, presented at SIGGRAPH this summer. All of it, including speaker notes, page by page, with a table in the engine’s documentation saying what each range of pages changes for Meadowbound, and what it does not.

The second half of that sentence is the point. Presentations like these contain history: prototypes the authors abandoned, resolution tricks they used on a console, alternatives they compared. “Use everything” cannot mean building all of it at once. One rule came out of the reading and binds everything since: there is one cloud field. The ground, the camera inside a cloud, the view from orbit, reflections and shadows all ask the same density function. A lower resolution cannot quietly substitute a different, cheaper cloud.

What was tried and thrown out

A better noise pattern. Instead of starting each ray’s steps at the same place, you can offset them with a carefully designed noise that changes every frame, so the history averages the gaps away. EA’s FAST sequences are made for exactly this. It went into the real Metal renderer, passed its tests, and was measured: over a cloud-covered aerial view the flicker, the same second difference as on the ground, went from 0.8070 to 2.2298. Fewer stripes, much more grain. It was removed.

It came back later, in a different form. The reconstruction that blends each new cloud sample with its history was rebuilt: it now estimates where in depth the cloud actually is along each ray and checks the history against that, instead of treating a noisy estimate as a solid surface. With that, the same view measured 0.3259, well below where it started. But it is also smoother, and smoother can mean less detail; that is written next to the number, and it is not a proof of zero flicker.

A jittered orbit. Seen from orbit, the clouds showed fine regular rings, the pattern of pixels against a pattern of clouds. Nudging each pixel’s ray inside the pixel broke up the rings, and doubled the flicker, from 2.6152 to 5.2928. Removed.

A light cache. If the sun is not moving, the walk towards it could be stored in a grid and reused. A 64 × 64 × 64 grid of 500-metre cells, rebuilt every frame, went in. Four matched runs later the cloud pass had gone from 107.905 to 106.732 milliseconds on the ground and from 145.191 to 145.102 in the air: no real gain, because most of the visible work was outside the grid. Removed.

Each of these is kept in full, with its source, tests and measurements, outside the engine. A failed experiment is information. A failed experiment left in the product is a bug.

What was kept

Clouds at every height they occur. The cloud layer used to be a fixed shell. Now the base and depth follow the local weather: low decks from 80 metres, ordinary cumulus higher up, storm towers above 20 kilometres, thin ice clouds between 6.5 and 16. A test checks 1,331 weather combinations against the bounds.

No cap on the ray. Cloud rays used to stop at 320 kilometres. From high up, looking at the horizon, a ray passes through the cloud layer twice: once near, and again on the far side, beyond the planet’s curve. The cap cut the far half off. It is gone. The ray now finds both stretches through the shell, skips the empty gap between them, and shares its steps by length: at least 224, one more for every kilometre of cloud it crosses, up to 4096. A fixed 224 was tried first; it missed the far band of a thin ice cloud, measured. The comment above the line says what it is, and what it is not:

// Long grazing paths need more quadrature than nearby cloud columns.
// A fixed 224 samples can miss an entire thin ice band on the far side.
// This bounded correctness budget is not a claimed performance win;
// hierarchical optical transport must replace its long-path cost.
const int kMaxSteps = clamp(int(ceil((firstLength+secondLength)/1000.0f)),224,4096);

The frame time

And now the number that is not good. At 2560 × 1440, looking down on the coast, the GPU currently needs a median of 373.8 milliseconds for a frame. The cloud pass alone is 128.9 of it, and the fog, light shafts and atmosphere scattering 205.2.

That is the next work: to make the clouds cheaper without making them worse. The papers say where the time goes, and the measurements say which of their ideas actually save it here. Everything else is a bet.