A revised forecast is relearned in as many crossings as the learner remembers

A season learns the stretch from where the streak was, not from the floors took nineteen forecasts of a jet streak drifting along the Goose Bay–Narsarsuaq gap, drawn at four fifths of the real spread, and asked how many past crossings it takes to learn the stretch that makes them honest. Each member’s starting position and drift speed is moved away from the ensemble’s centre by a factor λ, and at λ = 1.25 the members spread as widely as the truth and the highest member’s diversion floor covers the floor actually met in nineteen crossings of twenty — the condition an ensemble carries its own margin, if its spread is honest set for the plan to work at all. Learned from where the analysis after each crossing says the streak was, a season of a hundred crossings pins the stretch to within a tenth 85 times in a hundred.

That learner assumed one forecast system for the whole season. The stretch is a statement about the system — about how widely its members spread compared with its errors — and forecast centres revise their systems, typically once or twice a year, with new physics, new resolution, new ways of perturbing the members. A revision changes exactly the number the stretch repairs. The earlier essay ended by asking how quickly a stretch can be relearned after one, by an operator who weights recent crossings more heavily than old ones, and whether the bias that neither learner could see might be learned alongside it.

A revision mid-winter, and four ways to keep learning #

The setting is the earlier essay’s. Each crossing has a centre drawn uniformly over the 3,100 kilometres of jet from which a four-hour drift can bring the streak over the gap; the truth departs from the centre by 100 kilometres in starting position and 50 kilometres an hour in drift speed, as normal errors; and the plan is the highest of nineteen members’ floors, each member read from the same table of exact floors at thirty-five positions of the streak, the geometry a streak along the jet moves the floor while the crossing is flown introduced. The new element is time. A winter runs to 700 crossings: for the first 400 the members spread 0.8 of the truth’s spread, and for the last 300 a revised system spreads them at another width. Two revisions are measured — one that narrows the members to 0.6, so that the right stretch rises to 1/0.6 = 1.667, and one that widens them to the truth’s own spread, so that the right stretch falls to 1.

Every learner uses the earlier essay’s better statistic. After each crossing it takes the truth’s departure from the centre in each of the two numbers, divides by the members’ own spread about the centre on that crossing, squares, and averages; the root of the average, less the 17/19 inflation that dividing by an estimated spread brings, is the stretch. What differs is how the average weights old crossings.

The learner that never forgets weights every crossing since the start of the winter equally. The learner that restarts knows the date of the revision and begins a new average on it, keeping its old stretch until the new system has supplied a crossing. And the learners that forget weight each crossing by $(1 - 1/m)^a$ at age a crossings, an exponentially weighted average with a memory of m crossings, for m = 25, 50, 100 and 200. In steady state such a learner holds the information of about 2m − 1 crossings, so a memory of 50 is the forgetting equivalent of the earlier essay’s season of a hundred.

Every stretch is scored as the earlier essay scored it, and as the rule scored out of sample scored a rule of thumb, by the coverage it buys on crossings it was not learned from: 24,000 fresh crossings of whichever system is in force, the share of them whose floor the highest stretched member covers.

The learner that never forgets is still learning the old system #

Before the revision every learner holds 1.25 to within a few hundredths, and the system’s fresh crossings are covered 95.6 times in a hundred. At the revision nothing a learner holds changes, but the members do, and the same stretch now covers only 93.1 per cent: stretched by 1.25, members drawn at 0.6 of the spread still spread only three quarters as widely as the truth.

The learner that never forgets has four hundred crossings of the old system in its average when the first crossing of the new one arrives, and every new crossing adds one part in roughly five hundred. Its stretch climbs from 1.25 to 1.30 in fifty crossings, 1.34 in a hundred and 1.44 in three hundred — still 0.23 short of 1.667, having covered less than half the distance, and still the stretch of an average two fifths of whose crossings were drawn from a system that no longer exists. Over the two hundred crossings after the revision its plan covers 93.8 per cent of floors on average.

That is the cost of a learner that is excellent in a stationary season. In the earlier essay’s terms it was the right learner — the more crossings, the better the stretch — and the same property makes it the slowest to notice that the crossings have stopped being drawn from one system. Its average is a precise estimate of a mixture.

Coverage comes back after as many crossings as the learner remembers #

A forgetting learner’s average after t crossings of the new system holds a weight of $1 - (1 - 1/m)^t$ on them, close to $1 - e^{-t/m}$. The stretch it holds is the root of a weighted mean of the old system’s mean square, $1.25^2$, and the new one’s, $1.667^2$, and it comes within a tenth of the new stretch when that weight passes about 0.6 — which happens at t ≈ m. The measured recoveries agree with that arithmetic, and a little slower than it, because coverage is scored against 95 per cent rather than against the stretch: the mean coverage over 400 winters reaches 95 per cent and stays there from crossing 31 at a memory of 25, 57 at 50, 113 at 100 and 206 at 200.

So the earlier essay’s guess was close, and pessimistic by half in one respect. It expected a learner holding the equivalent of a hundred crossings to track a change with a lag of about a hundred. A learner with a memory of 50 holds that much in steady state, and it is back to 95 per cent in 57.

The restarted learner is faster again: its mean coverage reaches 95 per cent from the tenth crossing of the new system. A fresh average needs few crossings to move all the way, because nothing old holds it back; what it lacks at first is precision. After ten crossings its median stretch is 1.59, but the middle eight winters in ten span 1.28 to 1.93, and until the 37th crossing more than one winter in ten has a stretch that leaves coverage under 94.5 per cent. The mean recovers at once; any one operator’s coverage takes a month of daily crossings to become trustworthy, which is the earlier essay’s season of a hundred crossings appearing again from the other side.

Forgetting is paid for when nothing has changed #

A learner that forgets quickly pays for it when nothing has changed. In the hundred crossings before the revision, a stationary stretch in a stationary system, the learner with a memory of 25 holds a stretch between 1.13 and 1.39 in the middle eight winters of ten, and 3.7 per cent of winter-crossings have a stretch low enough to leave coverage under 94.5 per cent. At a memory of 50 that share falls to 0.4 per cent; at 100 and 200 to nothing measurable. The learner that never forgets and the one that restarts have, before the revision, four hundred crossings behind them, and never fall short.

So the two costs run in opposite directions along the memory, and they cross near fifty. Below it, a learner wanders often enough in a quiet season to be a cost in itself; above it, the recovery from a revision stretches past a hundred crossings — a quarter of a winter of daily flights spent under 95 per cent. A memory of fifty crossings spends 0.4 per cent of quiet crossings slightly short and recovers from a narrowing revision in 57.

That figure is not an accident of the setting. Forecast centres that correct their own ensembles for bias with a decaying average have tended to settle on weights of a few per cent for the newest forecast: the decaying-average bias correction applied to the North American ensembles by Bo Cui and colleagues at the National Centers for Environmental Prediction in 2012 put two per cent on the newest, which is a memory of fifty in the units here. The same compromise between responsiveness and noise lands in the same place because the noise of a squared departure is fixed by the statistic, not by the weather.

Restarting on the revision’s date beats every forgetting learner on recovery, and it needs one thing they do not: the date. Revisions are announced, so an operator who follows the forecast centre’s notices can restart, and one who does not, or who uses a product assembled from several centres revised at different times, cannot. Combining the two is the obvious operational rule — restart on an announced revision and forget at fifty otherwise — and it is a rule, not a measurement; the trials here measure each part.

A narrowing revision costs crossings, a widening one costs kilometres #

A winter’s account makes the asymmetry plain. Over the two hundred crossings after a narrowing revision, the learner that never forgets leaves 2.4 more crossings under the floor than a calibrated plan would; at a memory of 200, 1.3 more; at 100, 0.6. At a memory of 50 or less, and for the learner that restarts, the account is slightly in credit, because a learner on its way up from 1.25 overshoots a little in some winters and the coverage curve is flat above the right stretch.

A widening revision runs the other way. Members drawn at the full spread need no stretch at all, and a learner still holding 1.25 over-stretches them, which buys coverage rather than costing it: 96.7 per cent instead of 95.5 at the right stretch, and 3.2 fewer uncovered crossings than calibration over two hundred for the learner that never forgets. What it costs is allowance. On covered crossings the highest member sits above the floor actually met by 7.7 km on average at the right stretch, and by 8.4 km for the learner that never forgets — two thirds of a kilometre held in hand on every crossing that did not need it. That is not a negligible sum: the rule that keeps a route near land is pinned at both ends found that near its lower wall the first kilometre of allowance is worth twelve of route.

So the two directions of revision are not symmetric in what they risk. Under-stretching is a safety cost, counted in crossings below the floor; over-stretching is an economic one, counted in kilometres of allowance carried — and a crossing is a chain of decisions found that allowance fixed before departure, so every kilometre of it is carried the whole way. An operator who cannot tell which direction a revision went has a reason to forget faster than the quiet-season trade-off alone would suggest, because the crossing-count cost of a slow learner falls only on narrowing revisions, and those are the ones nobody notices until the crossings show it.

A shared bias is extra spread to one learner and a shift to another #

The earlier essay’s other loose end was a bias: a centre that misplaces the streak the same way on every crossing — the error a forecast’s error in where the streak is becomes a margin first priced, now made systematic — which a rank histogram shows only as a tilt and which the streak’s learner, squaring each departure, reads as spread. A revision is a natural place for one to appear, and the trials give it one: from the revision on, the truth starts 60 km short of the centre on every crossing, three fifths of the position error, while the members keep spreading 0.8 as widely as the truth.

The learner that only stretches does exactly what the earlier essay predicted. A departure that is 60 km on average plus its own error has a mean square larger than its variance, so the learner’s stretch rises from 1.25 to 1.33 after a hundred crossings and 1.36 after three hundred — the root of the pooled mean square, $\sqrt{(1.36 + 1)/2}/0.8$, to the hundredth. It covers a one-sided error with a symmetric stretch.

A learner that also tracks the mean departure separates the two. It keeps a forgetting average of each signed departure as well as of its square, shifts every member by the mean departure times the members’ spread, and stretches the members about the shifted centre by the spread of the departures about their mean. Its stretch stays within three hundredths of 1.25 throughout, and its shift in starting position is −42 km after a hundred crossings and −60 after three hundred, the middle eight winters in ten between −69 and −49. The shift is learned at the same pace as the stretch, because it is the same kind of average, and costs no crossings the stretch did not already need.

The bias costs half a point one way and nothing the other #

How much the bias matters depends on which way it points, because the floor is not symmetric in where the streak starts. With the truth 60 km short of the centre — the streak arriving at the gap later than forecast — a plan stretched by 1.25 with no shift covers 94.6 per cent of crossings, half a point short. With the truth 60 km beyond the centre it covers 95.9, more than a calibrated plan. The same shared error is a safety cost in one direction and none in the other: a streak that reaches the gap late and one that reaches it early are not mirror images.

Both learners recover the half point where it is lost. The stretch-only learner’s extra stretch brings the short-of-centre case to 95.0 per cent, the joint learner’s shift to 95.3, against 95.4 with the bias known exactly. Where the bias is harmless, the stretch-only learner over-stretches anyway, to 96.3 per cent coverage and 8.1 km of allowance on covered crossings, where the joint learner holds 95.8 and 7.2 km, the same as knowing the bias. The joint learner is right in both directions; the stretch-only learner is right where it matters and pays about a kilometre of allowance where it does not.

That settles the earlier essay’s question in a way it did not anticipate. A shared bias of this size can be learned alongside the stretch from the same analyses, in the same number of crossings, and a centre corrected and a spread stretched in one step is what the joint learner is. But the stretch alone was never badly hurt by it. Squaring the departures converts a bias into spread, and spread is what a plan built on the highest member needs more of when the truth falls outside the members on one side; the conversion is wasteful, not dangerous. It is the distinction averaging tracings reads the easing, until the hand repeats itself drew between an error shared by every measurement and one drawn afresh, and a week of twilights learns what a sight is worth drew between an evening’s shared refraction and each sight’s own: the shared part has to be estimated as a mean, not as a spread, if it is to be removed rather than covered.

What the two learners were held to #

Before the revision, every learner learns the same thing. Over the hundred crossings before it, the median stretch held by the learner that never forgets and by the one with a memory of 200 must lie within five hundredths of 1.25 in every crossing. It does, in both trials.

The revised system is the old one at a new stretch. Stretched by 1/0.6 and by 1, the revised members must cover 95 per cent of the new system’s fresh crossings, less two sampling standard errors. They cover 95.6 and 95.5.

The finding must be there to fail. After the narrowing revision, the learner that never forgets must still be more than a tenth short of the new stretch after 300 crossings. It holds 1.44 for 1.667.

The joint learner must find the bias. After 300 crossings its median shift must lie within a fifth of the bias. It finds 58.8 km for 60 beyond the centre and 60.0 for 60 short of it.

Every comparison is on one set of winters. Each winter’s departures are drawn once and every learner learns from the same ones; every stretch is scored on the same fresh crossings of the system in force.

Two revisions, one bias, and a known date #

Two revisions. The members narrow to 0.6 or widen to 1.0, in both numbers at once, on one crossing. A real revision may change the spread in starting position and drift speed differently, or be phased in over several cycles; the learner that pools the two numbers would then hold one stretch for two needs, which the earlier essay already flagged as a reason to keep them apart.

A bias in one number. The shared error is 60 km in starting position, constant from the revision on. A bias that depends on the season, or on where the streak is, is not constant across crossings, and a forgetting average would follow it only as fast as its memory.

The analysis taken as the truth. As in the earlier essay, the learners take the reconstruction after each crossing to say where the streak started and how fast it drifted. A revision of the analysis itself — the same centre’s assimilation system revised alongside its forecasts — would move the truth the learners measure against, and nothing here models it.

Crossings, not days. A memory of fifty crossings is fifty days on a route flown daily and a fortnight for an operator flying it four times a day; the trade-off is in crossings, and so is every number above.

Still open: whether a learner can tell a revision from a bad week #

A forgetting learner treats every change in the mean square of its departures alike, and two very different things change it. A revision changes it permanently. A run of crossings in an unusually unpredictable pattern — a blocking high, a jet that splits — changes it for a week and then puts it back. A learner with a memory of fifty follows both, which is right for the first and wasteful for the second: it stretches for a week of hard forecasts and then carries the extra stretch for weeks after they end.

The restarted learner shows the alternative. It knew the date, and that knowledge was worth forty crossings of recovery. A learner that did not know the date could look for it — test, at each crossing, whether the recent departures are better explained by a new stretch than by the old one, as a control chart tests for a shift in a process, and restart only when the test says the system has changed. Whether such a test detects a narrowing from 0.8 to 0.6 within the thirty or so crossings a restart needs to settle, how often it mistakes a bad week for a revision, and whether the tonight’s residuals problem of a short sample estimating its own air badly returns at the moment of detection, are questions a learner that forgets at a fixed rate cannot ask.