A week of twilights learns what a sight is worth

Weighted by what it disagrees with, the fix trusts the sight it leans on tried to learn each sight’s worth from one fix and failed in a particular way. A residual shows only the part of a line’s error the fix has not already absorbed, and the line the fix leans on hardest shows least. Reweighting by residuals therefore trusted a low body standing alone in its direction six times more than its altitude earned, and iterated to convergence it made the fix worse than no weights at all.

The weights that work, those of a low sight is worth keeping only if it is weighted, come from outside the sights: a sextant error of half a minute of arc, the same for every body, and a tenth of the refraction correction taken as uncertain, which near the horizon is several times the sextant’s error. The half minute is a property of the instrument and the observer. The tenth is a statement about the air, and it was stated, not measured.

A navigator takes sights at every morning and evening twilight, and the low bodies are low again and again. The pooling has a precedent on this ground: what a fifth unknown costs when nothing can see it solved for a common error in every sight as one more unknown and found it estimable only where the geometry let it be seen. A variance is a harder thing to see than an error, and pooling is how more geometry is brought to bear on it. What one set of sights cannot say, a week of them might, because the leverage that hid a sight’s error in one fix is different in the next. This essay asks how many twilights it takes to learn the share, and then asks the question the first one hides: what knowing it is worth.

Every residual is a point on one line #

The residuals of a fix are the errors of its sights passed through one matrix, the residual operator M = I − H of the equal-weight fit, so line i’s residual is $\sum_j M_{ij} e_j$. The sights’ errors are independent, so its expected square is exact:

$$\mathrm{E}[r_i^2] \;=\; \sum_j M_{ij}^2\,\sigma_j^2 \;=\; \sigma_0^2 \sum_j M_{ij}^2 \;+\; k^2 \sum_j M_{ij}^2 R_j^2,$$

where $\sigma_0$ is the sextant’s error, $R_j$ the refraction correction at sight j’s altitude and k the share of it that is uncertain. Both sums are computed from the fix’s own azimuths and altitudes before anything is observed — the same kind of number the residual reports the error the fix was immune to computed from four azimuths to say how much of a common error a residual could show. So every residual of every fix is one noisy observation of a straight line in two unknowns, $\sigma_0^2$ and $k^2$, and a week of twilights is a regression.

The leverage is inside the predictors. The low sight that carries the fix’s geometry has a small $M_{ii}$, and its residual’s expectation says so; what it can still reveal of its own error appears in its own predictor, and what it spreads into its neighbours’ residuals appears in theirs. That is the correction for leverage done exactly, rather than by dividing each squared residual by $1 - h_i$, which is what the one-fix scheme tried and which is right only for the line’s own share. The fit is least squares of the squared residuals on their two predictors, reweighted once by each one’s expected variance, and a negative estimate of $k^2$ is reported as a share of zero.

Drawn that way, the residuals of four weeks lie scattered about a line whose intercept is the sextant’s variance and whose slope is the square of the share. The scatter is large because a single squared residual is a single draw from a distribution as wide as its own mean. The binned means sit close to the line, and the fit returns a share of 0.089 and a sextant error of 0.50 minutes from 336 residuals, against a truth of a tenth and half a minute.

The fit’s slope is decided almost entirely by the residuals on the right, and they belong to the low bodies. A body at forty-five degrees has a refraction correction of about a minute, and its uncertain tenth is a fifth of the sextant’s error, which adds in square to almost nothing. A body at five degrees has a correction of ten minutes, and a tenth of that is twice the sextant’s error. The share can be learned only from sights for which it matters.

How many twilights #

A fix here is six bodies on random azimuths spanning at least a third of the compass, at random altitudes between 4° and 70°, which is what a navigator choosing from a star list at a twilight gets rather than a designed layout. A second navigator does the same and always includes one body between 3° and 8°. For each, 1,500 seeded weeks are run, and the share is estimated after every count of twilights from one to fifty-six.

One twilight teaches almost nothing. Six residuals, four degrees of freedom, and the low body’s error mostly absorbed into the fix: the estimate’s middle eighty per cent runs from zero to 0.181, and its median is half the truth for the navigator with a low body and a fifth of it for the one without. A week of fourteen twilights narrows the middle eighty per cent to between 0.055 and 0.131 with the median at 0.092. Four weeks narrow it to between 0.079 and 0.116.

The medians climb to the truth from below rather than scattering about it, and that is the square root at work. The regression estimates $k^2$, which it gets right on average, and the share is its root. A root of a noisy quantity centred on 0.01 is pulled down whenever the noise is large, and the negative draws, which are common early, are set to zero.

The zeros are the practical danger. After one twilight the navigator with a low body gets an estimate of exactly zero 39 times in a hundred and the other navigator 49 — the residuals say the refraction correction is perfect. A navigator who took that at face value would weight the low sight as though it were as good as a high one, which a low sight is worth keeping only if it is weighted showed is worse than discarding it. By a week the zeros have fallen to 2 per cent with a low body and 12 without. Within half again of the truth: 78 per cent of weeks with a low body, 58 without.

A week is therefore the answer to the question the one-fix measurement left, for a navigator who takes a low body: a week of twilights learns the share to within half again of the truth four times in five. For one who does not it takes about four weeks to reach the same confidence, and the reason is the same line through the same points — without low sights there are no points far enough to the right to fix its slope.

The fix is flat around the right share #

The first half of the question has an answer. The second half is what the navigator gets for it.

A fix’s error depends on its weights, but not steeply near the best ones. The best linear fix is the one weighted by the true variances, and a small change of weight about that optimum changes the fix’s error only in second order — which is Gauss’s theorem stated as a shape rather than a bound. Assuming a share of 0.05 or 0.2 when the truth is 0.1 costs the navigator with a low body 3.8 and 3.7 per cent of the root-mean-square fix error. Assuming 0.4 costs 12.7, and 0.8 costs 23. The curve is a shallow valley with steep walls a long way off.

Its depth is set by the low body. Weighting every sight equally — assuming a share of zero and a sextant error that is the same for every body — costs the navigator who always takes a low body 15.4 per cent, and the one who does not 5.5. The prize for weighting by altitude at all is those numbers. The prize for knowing the share, as against guessing it within a factor of two, is at most four points of them.

The fix error here is averaged over twilights. Weighted by what it disagrees with found that for particular arrangements — a low body alone in its direction — the stakes are larger, because the fix leans on that line and its weight matters more. A navigator whose sky that evening happens to be such an arrangement gains more from the right share than the average says. Averaged over the skies a navigator actually meets, the valley is as flat as it is drawn.

What the learning is worth #

A navigator who weights each fix with the share learned so far pays 4.2 per cent after one twilight, which is slightly worse than a sensible guess because an estimate of zero four times in ten is a bad guess. After a week the cost is 1.27 per cent, and after four weeks 0.22. The navigator without a low body pays less throughout — 0.63 per cent after a week — because there was less to lose.

So the week’s learning is worth about two and a half points of fix error over a guess within a factor of two, for the navigator who takes a low body, and one or two points for the one who does not. The best fix from these six sights is good to about a kilometre, root-mean-square, so two and a half points is some twenty-five metres. It is real and it is small, and it is far smaller than the error a navigator takes on by guessing the share wrong by a factor of four or more, or by not weighting at all.

The same comparison says what the stated tenth was worth. Every weighted fix before this one assumed a share of a tenth and called it stated. If the truth had been a twentieth or a fifth, those fixes were a few per cent worse than they could have been, and no conclusion drawn from them — that a low sight must be weighted, that residual weights trust the leveraged sight — depended on the tenth being right. The steep walls of the valley are where those conclusions live, and the stated tenth is on the flat floor.

The body near the horizon does both jobs #

The two navigators differ by one habit, and it decides both halves of the account. The low body is why weighting matters: without it, every sight has much the same variance and weights change the fix by five per cent at most. It is also why the share can be learned: without it, no residual sits far enough along the line to fix its slope. A navigator who avoids low bodies needs weights less and can learn them less, and the two effects are the same fact about where the refraction error lives.

That is the opposite of the one-fix result in a useful way. There, the low body alone in its direction was the sight whose residual lied most, because the fix had moved to meet it. Across many fixes the same body is the one that teaches, because its leverage changes from twilight to twilight as the stars move, and the pooled regression accounts for every one of those leverages exactly.

A share that changes nightly #

Everything so far assumed one share for the whole week, which is what a settled air mass would give. The share is a statement about how far the air near the horizon departs from the standard atmosphere a refraction table assumes, and that departure changes with the weather. Near a cold sea under warm air it can change by a large factor from one evening to the next.

Pooling learns the typical share and nothing about tonight’s. A navigator who knew the typical share exactly and weighted every twilight with it pays 0.4 per cent when the nightly share varies by a quarter, 1.3 when it varies by half, and 4.3 when it varies by a factor of two. The last is more than the whole of what a week’s learning bought over a guess. So the learning is worth having only where the air is steadier than a factor of about one and a half from night to night, and where it is not, the dominant error in the weights is the night’s own weather, which no amount of pooling can see.

The comparison with equal weights survives all of it. At a nightly factor of two, weighting with the typical share costs 4.3 per cent and weighting nothing costs 30.8. Whatever the share does, a navigator who weights by altitude with a roughly right share is far ahead of one who does not, and the details of the share are a correction to that, not a replacement for it.

Who worked out the estimator #

Estimating the variances of groups of observations from their residuals is older than modern statistics. Friedrich Robert Helmert gave a method for it in the second edition of his treatise on least squares in 1907: split the observations into groups with different unknown variances, compute what each group’s weighted sum of squared residuals would be expected to be in terms of those variances, and solve the resulting equations. C. R. Rao’s minimum-norm quadratic unbiased estimation of 1970 is the general form, and in geodesy it is called variance-component estimation. The weights are a guess the solve believes met it inside a single network, where the groups were two kinds of observation.

What is done here is the same thing with a continuous model rather than groups. Each sight’s variance is $\sigma_0^2 + k^2 R^2$, linear in two unknowns, so the expected squared residuals are linear in them too, and pooling across fixes is only a matter of stacking the equations. A single fix of six sights gives six such equations with four degrees of freedom among them, which is why one twilight fails; fourteen give eighty-four.

How the estimate was checked #

The expectation must be exact. For one stated twilight, the mean squared residual of each line over 20,000 draws of its sights’ errors must equal $\sum_j M_{ij}^2 \sigma_j^2$, and does, every line within 5 per cent — the sampling scatter of 20,000 squared normal variables.

Noise-free, the fit must return the truth. Given squared residuals equal to their expectations, the fit returns a share of a tenth and a sextant error of half a minute to six decimal places.

No refraction error must give no share. With the true share set to zero, twenty-eight twilights’ pooled estimate has a median under 0.03 — above zero only because a root of a noisy variance cannot be negative.

The weights’ cost is exact per twilight. It is the trace of the fix’s covariance, the quantity whose shape a confidence ellipse is honest only where the sheet keeps angles drew, and it is computed here in the plane of the fix, where no sheet intervenes. The fix error for given weights and true variances is the sandwich covariance (AᵀWA)⁻¹AᵀWVWA(AᵀWA)⁻¹, computed rather than sampled for each of 3,000 seeded twilights, so the averages carry only the scatter of which skies were drawn.

Where the account stops #

Random skies. Six bodies on random azimuths and altitudes is a model of a star list, not of a navigator’s choice. A navigator who chooses bodies to spread round the compass makes the weights matter less and the residuals more informative about everything but the low bodies.

The sextant’s error is one number. An observer’s error rises with fatigue, sea state and a hazy horizon, and it can be learned in the same regression with more predictors. Every term added costs degrees of freedom the week has to supply.

The refraction model is Bennett’s table and a share of it. Real anomalous refraction near the horizon is not a fixed fraction of the standard correction; a strong inversion bends low light in ways a single share cannot describe, and the lowest sights would then carry an error that this model calls noise.

The fixes are independent. A ship moving between twilights is fixed from a running fix and dead reckoning, and an error in the run between two twilights enters both. The regression treats each twilight’s residuals as its own.

Still open: whether the residuals can see tonight’s air #

The nightly variation is the result that decides the practical question, and it points at a harder measurement. Pooling across nights learns a typical share because it treats every night alike. A navigator does not need the typical share; the navigator needs tonight’s, and tonight’s is exactly what one twilight’s six residuals cannot deliver — the estimate is zero four times in ten.

Two things could change that. A twilight with more than one low body gives the regression more than one point on the right of the line, and a navigator who took three bodies below ten degrees might learn something about the evening’s air from the evening alone. And the weather itself is observed: the difference between the air and sea temperatures is what drives the anomaly, and a share modelled as a function of that difference would pool across nights by what they have in common rather than by assuming them the same. Whether three low bodies in one twilight estimate the evening’s share well enough to beat the typical one, and whether a share predicted from the air-sea temperature difference beats both, are questions a regression that pools every night alike cannot ask.