Green screen tolerance, explained

Every chroma key has a tolerance control and almost none say what it does. This one moves a single distance threshold — which is why turning it up can rescue a badly lit screen or quietly erase your subject.

The one number it moves

Tolerance does not widen the green range in any visual sense. It sets a radius. The slider maps onto a threshold called lo:

lo = (tolerance / 100) × scale

The scale is 118 for a green or blue screen and 150 for a neutral backdrop — a white or grey wall, where there is no chroma to measure and the keyer falls back to luminance. At the default tolerance of 25 that is 29.5 for a chroma key and 37.5 for a neutral one. Those are not percentages or degrees; they are distances in the colour plane, and everything below uses the same units.

Distance from what, measured how

Each pixel is converted to YCbCr with the BT.601 coefficients, and only Cb and Cr are kept. Brightness is discarded on purpose: a shadow across the screen is the same paint with less light on it, and in RGB that moves the pixel a long way even though its colour has not changed.

The distance is the straight-line gap between the pixel's colour point and the key colour's point. At or inside lo a pixel is background and gets alpha 0; at or beyond a second threshold hi it is fully opaque; in between, alpha is partial. With the edge control at 0, hi equals lo, so there is no soft band — tolerance alone gives a hard cut. Softening is the edge control's job.

When turning it up helps

A shadowed screen is the one case where raising tolerance is correct. On shadow-green — a soft shadow across part of the backdrop, same hue, less light — the score at the default is IoU 0.617718, with 21,201 backdrop pixels surviving. Raise tolerance and it climbs:

• tolerance 25 → IoU 0.617718 (21,201 backdrop pixels left behind)
• tolerance 30 → IoU 0.671108 (18,240)
• tolerance 35 → IoU 0.770641 (12,720)
• tolerance 40 → IoU 1.000000 (clean)

The subject sits further out than the shadowed backdrop, so nothing is lost on the way.

When turning it up eats the subject

The threshold does not know what is backdrop and what is subject. It only knows distance. The subject in these fixtures is a skin tone about 105.94 units from the green key colour. At tolerance 85 the radius is 100.3, still short of it. At tolerance 90 it is 106.2 — past the subject — and on flat-green all 21,341 subject pixels are keyed out. IoU drops from 1.000000 to 0.722122.

A neutral backdrop fails sooner, the scale being larger. A grey prop on a near-white wall sits 88 units of luminance away. The radius passes that at tolerance 60 (0.60 × 150 = 90), and from there the whole subject goes transparent: IoU 0.645560, unchanged to the top of the slider.

The spill case: a peak, not a direction

On spill-green, where light bounced off the screen onto the rim of the subject, the cost is graduated. IoU is 1.000000 at tolerance 25, then 0.991136 at 30 (496 subject pixels gone) and 0.978579 at 35 (1,214 gone). The rim is closer to the key colour than the rest of the subject, so it goes first; the spill page covers why it exists.

With both problems at once (shadow-spill-green, plus sensor grain) the shape is obvious: 0.622672 at 25, 0.869190 at 35, 0.966269 at 40 (1,936 lost), 0.955053 at 45 (2,610 lost), 0.777652 at 90 (15,857 lost). The best result sits at 40 and going past it makes things worse: tolerance has an optimum, it is not a control you push until the screen looks clean.

Why blue survives more of it

That skin tone sits 118.19 units from the blue key colour rather than 105.94. On flat-blue the keyer still scores IoU 1.000000 at tolerance 100, where the radius is exactly 118.0 — a margin of 0.19 units. That is arithmetic luck at the top of the slider, not a guarantee: another subject colour would not get it, and it says nothing about which colour to shoot on.

How to set it on a real clip

• Start at the default 25 and judge the result on the backdrop, not the subject.
• Raise it a notch at a time and stop the moment the last of the screen is gone. Every notch past that only buys a thinner subject.
• If you need past roughly 40 to clear shadow, the shadow is the problem. Relight instead — that is what the lighting guide is for.
• Tolerance is one of three controls: it decides what counts as backdrop, edge decides how the boundary is feathered or trimmed, spill pulls key colour out of the subject. Moving one without watching the others is where bad keys come from.

Where these numbers come from

Every figure comes from data/tolerance-sweep.csv in the chroma-key-reference repository, and the method is written up at DOI 10.5281/zenodo.22916767. The fixtures are synthetic: 320×240 flat colour fields with an ellipse subject in a skin tone, generated deterministically so the sweep is reproducible. They are not real footage. A real clip has grain, compression, motion blur and a subject of many colours, so read these numbers as the shape of the behaviour, not as settings to copy.

When this does not work

• A subject close to the screen in colour. A green shirt on a green screen is not a tolerance problem. Any radius that reaches the screen reaches the shirt.
• Deep shadow or a badly uneven screen. Tolerance absorbs a moderate falloff — the shadow fixture cleans up at 40 — but past that the radius reaches the subject first. Relight the backdrop.
• Transparent or semi-transparent subjects: smoke, glass, water, flyaway hair over the screen. Distance gives one answer per pixel, and a pixel that is half backdrop and half hair does not have one.
• Compression artefacts at the edge. Chroma subsampling smears the screen's colour into the edge, so the artefact sits close to the key colour and raising tolerance eats edge with it.
• A neutral backdrop with a mid-grey subject. The luminance fallback has no chroma to work with, and 88 units of luma is thin margin once the wall is unevenly lit.
• Motion blur. A blurred edge is a blend of subject and screen, and each blend value maps to a different distance. No single radius treats the band uniformly.
• Trying to fix spill with tolerance. It removes the rim by deleting those pixels, which shows up as a chewed edge in hair. Spill suppression changes colour; tolerance deletes pixels.
• Pushing the slider to the top. Nothing in the sweep improves all the way to 100: the best scoring tolerance here is 40, and the flat cases collapse at 60 or 90.