A season learns the stretch from where the streak was, not from the floors

An ensemble carries its own margin, if its spread is honest planned the ETOPS diversion floor for the New York–London chain from nineteen forecasts of a jet streak drifting along the Goose Bay–Narsarsuaq gap. If the truth is statistically one more member, the highest of the nineteen members’ floors covers the floor actually met in nineteen crossings of twenty, whatever the forecast’s error, and it holds a fifth to two fifths less allowance in hand than a fixed margin sized for the same coverage. The condition is the spread. An ensemble whose members scatter less widely than the forecast’s real errors puts the truth above all of them too often, and nothing inside one crossing can say so.

The repair it named is standard in weather forecasting: stretch every member away from the ensemble’s centre by a factor learned from past crossings. What it left open was how many past crossings that takes, given that each crossing seems to contribute only one rank — where the truth’s floor fell among the members’ — and that in a quarter of crossings the streak never reaches the gap and every floor is the same to within a kilometre.

The answer depends on what a crossing is asked to contribute. Asked for a rank, a season has to be very long. Asked for the thing the rank was a consequence of, it does not.

Four fifths of the spread, and a stretch of a quarter #

The setting is the earlier essay’s, with its failure built in. Each crossing has a centre — the ensemble’s best estimate of where the streak starts — drawn uniformly over the 3,100 kilometres of jet from which a four-hour drift can bring the streak over the gap. The truth departs from the centre by a normal error of 100 kilometres in starting position and 50 kilometres an hour in drift speed. Each of the nineteen members departs by four fifths of that, in both numbers. The floor each member meets is the worst over its own crossing window, read from the same table of exact floors at thirty-five positions of the streak, and the plan is the highest member’s floor.

A stretch λ moves every member’s start and drift speed away from the centre by the factor λ. At λ = 1.25 the members spread exactly as widely as the truth, because 1.25 undoes four fifths. The figure is the price of each stretch, measured on 24,000 crossings drawn afresh: 94.1 per cent coverage unstretched, past 95 between 1.1 and 1.15, 95.8 at the calibrating stretch, and a slow climb beyond it as the members become too wide and the highest of them sits further above the truth.

Coverage is the score because the floor is a wall rather than a target: the rule that keeps a route near land is pinned at both ends found that a plan meeting it on average is on the wrong side of it in half the crossings, and a crossing is a chain of decisions found the allowance fixed before departure, so whatever the ensemble misjudges is paid for in the plan and not repaired in the air.

Two things in that curve set the scale of the problem. The coverage at λ = 1 is 94.1 per cent here against 93.7 in the earlier trials, which drew a different twelve thousand crossings; the difference is sampling, and it is the same size as the shortfall being repaired. And the coverage is flat enough near 1.25 that a stretch of 1.15 or 1.35 would serve almost as well. A season does not need to find 1.25 to a hundredth. It needs to land within about a tenth, and it needs to do so reliably, because a season that learns 0.95 leaves the ensemble narrower than it found it.

A rank histogram from one season #

The standard check on an ensemble’s spread is the rank histogram, introduced for weather ensembles in the mid-1990s: over many forecasts, count where the truth fell among the members. Calibrated, the truth is equally likely to fall in any of the twenty places, and the histogram is flat. Too narrow, and the truth falls outside the members too often, so the two end bins pile up.

One season of two hundred crossings shows the pile and shows how faint it is. The two end bins hold 35 crossings, where a flat histogram would put 20 there — the ensemble’s narrowness, visible. The other eighteen bins hold the other 165 and wander between 2 and 14 by chance alone, so that one interior bin, with two crossings in it, is as far from the flat line as the ends are in the other direction. A reader handed this histogram without being told the answer could see that the ensemble is too narrow, and could not say by how much.

The lighter part of each bar is the other thing the earlier essay pointed at. In 42 of these 200 crossings, and in 25.6 per cent of crossings over a long run, the streak cannot reach the gap during the four hours whatever the error, and every member and the truth meet the same floor to within a kilometre. Their order among themselves is decided by metres, which is to say by the smooth interpolation between table entries rather than by anything an operator would ever file. The trials here leave those ranks in, which is the floors’ best case: an operator with floors filed to the kilometre would have a quarter of the season as exact ties and learn nothing from them at all.

Two ways to learn the stretch #

The floors’ way is the rank histogram’s. For a stretch λ, re-rank the truth’s floor among the members’ floors stretched by λ, count the crossings in which the truth falls outside every member — above all of them or below all of them — and take the smallest stretch at which that count is no more than its calibrated share, two crossings in every twenty. The rule uses nothing a crossing does not report about its floors, and it targets exactly the property that makes the plan work: the truth landing above every member no more often than it should.

The streak’s way uses something the floors are a consequence of. After a crossing, the analysis — the forecast centre’s own best reconstruction of what the atmosphere did — says where the streak actually started and how fast it actually drifted. Every member also has a start and a drift speed. So the truth’s departure from the ensemble’s centre can be measured in units of the members’ own spread about that centre, separately for start and drift. Calibrated, those departures have a mean square of one; at four fifths of the spread they have a mean square of $1.25^2$, whatever the floor did. The stretch is the root of the mean square, pooled over both numbers and the season.

One correction belongs in it. Each crossing’s spread is itself estimated from nineteen members, and dividing a squared departure by an estimated variance inflates it, on average, by the factor M/(M − 2) — 19/17 here. Taken naively, the streak’s learner settles at a stretch of 1.32 on long seasons rather than 1.25. Multiplied by 17/19 it settles at 1.25, and every number below is corrected that way.

The two learners estimate the same number. The miscalibration is in the members’ starts and drift speeds, and the stretch acts there; the floors only carry its consequences. What differs is how much of each crossing each learner can use. The floors’ learner hears from a crossing only whether the truth fell outside the members, an event that happens to about one crossing in ten and says nothing about how far outside. The streak’s learner hears from every crossing a squared departure in two numbers, including the quarter of crossings in which the floor did not move.

A hundred crossings against eight hundred #

Four hundred seasons were drawn at each length, from 25 crossings to 800, every one learned both ways from the same crossings.

The floors’ learner is slow and it is low. At twenty-five crossings the middle eight seasons in ten learn anything from 0.78 to 1.50 — a stretch that narrows the ensemble further is as likely as one that overcorrects it. At a hundred crossings the range is still 0.95 to 1.35, and the median 1.13. It narrows steadily, and at eight hundred crossings it is 1.13 to 1.28 about a median of 1.20.

The low median is a property of the rule rather than of the floors. The rule takes the first stretch at which a season’s end bins fall to their share, and a season’s count of crossings outside the members wobbles as the stretch rises, so the first stretch at which it dips below the line is reached early more often than late. At 1.20 the fresh coverage is 95.3 per cent, so the bias costs nothing practical by eight hundred crossings. It does show that a rule built on a rare event inherits that event’s noise twice: once in how few crossings carry it, and again in where the rule decides to stop.

The streak’s learner is centred from the start. Its medians lie between 1.24 and 1.25 at every season length, and its range is narrower at twenty-five crossings, 1.08 to 1.43, than the floors’ range at a hundred. At a hundred crossings it is 1.17 to 1.34; at two hundred, 1.19 to 1.31.

How often a season pins it #

The question the earlier essay asked was whether a season pins the stretch to within the tenth that separates the coverage of a narrow ensemble from the coverage it should have. Counted that way, the comparison is stark. From the floors, a season of a hundred crossings lands within a tenth of 1.25 in 32 per cent of seasons, two hundred in 43, and eight hundred in 67. From the streak, a hundred crossings land within a tenth in 85 per cent, two hundred in 98, and four hundred in all but a few.

A hundred crossings learned from the streak pin the stretch more often than eight hundred learned from the floors. On this measure the streak is worth more than eight times as many crossings, and the ratio grows as the requirement tightens.

The reason is arithmetic, not luck. A learner that averages a squared departure over S crossings in two numbers estimates $\lambda^2$ with a relative standard error of about $1/\sqrt{S}$, so λ to about $1/(2\sqrt{S})$ of itself: at a hundred crossings, five per cent, or about 0.06 on 1.25. The middle eight seasons in ten then span 1.17 to 1.33, and the measured range is 1.17 to 1.34. A learner that counts an event happening to one crossing in ten has, in a hundred crossings, about ten events to count, and a count of ten is uncertain by a third of itself. The floors were never short of crossings. They were short of crossings that said anything.

What each season buys on crossings it has not seen #

A stretch is worth exactly the coverage it buys on crossings it was not learned from — the same discipline the rule scored out of sample applied to a rule of thumb — so every season’s stretch was scored on the 24,000 fresh crossings behind the first figure. The share of seasons whose stretch leaves the ensemble covering less than 94.5 per cent — worse, by half a point, than the plan was meant to deliver — is the practical failure.

From the floors, 47 per cent of 25-crossing seasons fail that way, 33 per cent of 100-crossing seasons, 16 per cent at two hundred, 6 at four hundred and 2 at eight hundred. A third of the operators who learned their stretch from a hundred crossings’ floors end up with an ensemble no better calibrated than the one they started with, and some with a worse one: the bottom of the floors’ range at a hundred crossings covers 93.5 per cent, below the unstretched ensemble’s 94.1.

From the streak, 7 per cent of 25-crossing seasons fail, 2 per cent at fifty, one season in four hundred at a hundred, and none from two hundred on. The coverage bought settles between 95.3 and 96.0 per cent by a hundred crossings, the narrow band the calibrating stretch’s own neighbourhood allows.

A transatlantic route flown daily makes a hundred crossings in a little over three months, and an operator flying several makes them in weeks. Learned from the floors, a season’s worth of crossings is too few to trust the stretch it gives. Learned from the streak, it is more than enough.

The quantity the plan is about is the wrong one to learn from #

It is natural to calibrate on the floor. The floor is what the plan is for, the ensemble’s job is to cover it, and the rank histogram of the floor tests exactly the property the plan depends on. Calibrating on anything else feels like changing the subject.

But the floor is a many-to-one summary of the streak. Every crossing in which the streak cannot reach the gap collapses every member and the truth onto one floor, whatever their starts and drift speeds; every crossing in which it can still reduces two numbers to one and the members’ positions to an order. The stretch is a statement about the streak’s starts and drift speeds, and those are reported in full by the analysis after every crossing. Learning the stretch from the floors throws that report away and keeps only what survives the floor’s summary of it.

The same shape has appeared elsewhere in this subject. A week of twilights learns what a sight is worth learned a refraction share from residuals corrected exactly for each fix’s geometry rather than from the fixes’ final errors, which mix the share with everything else the geometry did. The weights are a guess the solve believes found a variance estimable from the residuals of one group and invisible in the solution’s summary. Averaging tracings reads the easing, until the hand repeats itself found the limit on the other side: a spread learned from the members of a stack measures only what varies within the stack. The streak’s learner here has the same limit. It measures the truth against the members’ spread about the centre, so a bias every member shares — a centre that is wrong in the same direction every crossing — is invisible to it, as it is to the rank histogram’s end bins, and shows only as a tilt in the histogram.

Checks the learners were held to #

The fresh curve is the earlier trials’ curve. Unstretched, nineteen members drawn at four fifths of the spread must cover between 93 and 94.5 per cent of fresh crossings; stretched by 1.25 they must cover 95 per cent less two sampling standard errors. They cover 94.1 and 95.8.

Both learners find the stretch given enough. Over seasons of eight hundred crossings, the floors’ median must lie within a tenth of 1.25 and the streak’s within three hundredths. They lie at 1.20 and 1.25.

The finding has to be there to fail. At a hundred crossings the streak must pin the stretch within a tenth more often than the floors do. It does, in 85 per cent of seasons against 32.

Every comparison is on one stream. Seasons are drawn from one pool of 40,000 crossings, 400 at each length, and each season is learned both ways from the same crossings; every stretch is scored on one separate stream of 24,000.

Where the season stops #

One miscalibration, and the right kind. The members are too narrow by the same factor in start and drift speed, and the stretch is a single factor about the centre, so the repair has exactly the shape of the fault. An ensemble too narrow in drift speed and calibrated in start would need two stretches; the streak’s learner measures the two numbers separately and could supply them, and the floors’ learner could not.

The analysis taken as the truth. The streak’s learner assumes the reconstruction after each crossing says where the streak started and how fast it drifted. A real analysis has its own error, which adds to the departures and makes the learned stretch too large by however much the analysis error adds to the spread. Nothing here measures that, and it is the one way the streak’s learner can be wrong that the floors’ learner cannot.

Seasons with one jet. Every crossing has the same streak, a hundred kilometres an hour on a core of a hundred, and the same four-hour window. A season of weaker summer jets has more quiet crossings, which leaves the floors’ learner less to count and the streak’s learner untouched — so the gap measured here should widen in summer — but that is reasoning, and the summer season was not drawn.

A shared bias is invisible to both. A centre that misplaces the streak in the same direction every crossing moves the truth outside the members in the same direction every time; the rank histogram shows it as a slope, and the streak’s learner, which squares the departures, reads it as extra spread and stretches symmetrically to cover a one-sided error.

Still open: whether the stretch learned in one season serves the next #

A stretch learned from where the streak was is a property of the forecast system rather than of the weather, so it ought to carry from one season to the next. But forecast systems are revised, often annually, and the spread of a new version is the thing that has to be learned again. The useful question is how quickly a stretch can be relearned after a revision, by an operator who weights recent crossings more heavily than old ones.

That is the earlier question one step on. A season of a hundred crossings learned from the streak pins the stretch to a tenth 85 times in a hundred; a forgetting learner that keeps the equivalent of a hundred crossings at any time would track a change in the spread with a lag of about the same length. Whether that lag is short enough to matter within a winter, and whether the shared bias that neither learner sees could be estimated alongside the stretch from the same analyses — a centre corrected and a spread stretched in one step — are questions a single stationary season cannot ask. A forecast’s error in where the streak is becomes a margin and a streak along the jet moves the floor while the crossing is flown are where the floor and its forecast were first priced, and where a shifted centre would first show.