A low sight is worth keeping only if it is weighted

A confidence ellipse is honest only where the sheet keeps angles drew the region a celestial fix is likely to be in and measured what a plotting sheet does to it. Every sight in that essay, and in the four before it, had the same error. It is the assumption that makes the arithmetic of a fix look like geometry alone: lines of position, their azimuths, and a normal matrix built from nothing but directions. A cocked hat holds the ship one time in four needed nothing else, and neither did any of the regions drawn since.

Real sights are not like that, and the reason is the air. A body high in the sky is seen through a short column of atmosphere and its light bends by a minute of arc or less. A body a few degrees above the horizon is seen through a long, low, layered column, bends by ten minutes or more, and bends by an amount the tables can only estimate. This essay gives each sight the error its altitude earns and measures three things that follow: whether a low sight helps the fix at all, what the ellipse looks like when it is weighted, and what it costs when the weights are wrong.

What a sight’s error is made of #

A sextant altitude carries two kinds of error that behave differently with altitude.

The first is the instrument and the person: the index error left after correction, the reading of the arc, the judgement of when the body’s limb touches the horizon, the roll of the deck. None of these cares much how high the body is, and together they are commonly put at a few tenths of a minute of arc for a careful observer in fair conditions. Here that part is stated as half a minute, which is 0.93 kilometres along the body’s azimuth, since a minute of altitude is a nautical mile.

The second is the refraction correction. Light from a body is bent downward as it passes through air of increasing density, so the body appears higher than it is, and every sight is reduced by a tabulated correction. The standard formula for that correction at an observed altitude h, due to G. G. Bennett in 1982, is

$$R = \cot\!\left(h + \frac{7.31}{h + 4.4}\right)\ \text{arcminutes},$$

and it is 0.99 minutes at forty-five degrees, 5.39 at ten and 14.3 at three. The almanac tables are computed for a temperature of ten degrees Celsius and a pressure of 1,010 millibars, with a further table to adjust for others, and near the horizon the real atmosphere departs from any smooth model: a temperature inversion over cold water, a warm layer over a hot deck, a gradient the table knows nothing about. The correction is least certain exactly where it is largest.

So each sight’s standard deviation is stated as $\sigma^2 = \sigma_0^2 + (kR)^2$, the sextant’s half-minute and a share k of the refraction correction. The share is not measured here and cannot be; it depends on the weather. It is carried as a parameter, a tenth by default and swept from a fiftieth to a half.

With a tenth of the correction uncertain, a sight above thirty degrees is within six per cent of the sextant’s own error and the altitude hardly matters. Below ten degrees it matters more every degree: the sight at five degrees is more than twice as uncertain as one at forty-five, and at three degrees three times. With a fifth uncertain, the three-degree sight is more than five times the high one. The curve is flat and then it is steep, and the steep part is where every body near the horizon lives.

Unweighted, a sight below seven degrees is worse than no sight #

The fit in the essays so far is ordinary least squares: every line of position pulls on the fix equally. With a low sight among high ones, that means the worst line is trusted as much as the best, and it raises a practical question with a clean answer. Is the fix better with the low sight in it or without it?

The layout is three high bodies on one side of the ship, at azimuths of 35, 75 and 115 degrees, and a fourth behind at 190. The three high ones alone leave the fix poorly constrained along the direction they share, so the body behind is geometrically valuable: it is the only line crossing the others at a good angle. The question is whether its geometry outweighs its noise.

The measure is the root-mean-square distance of the fix from the ship, which weights a large miss more than a small one; the average was a choice of norm is the warning that another summary could rank the fits differently, and here the mean distance ranks the three fits in the same order at every altitude checked, with the crossing below moved by less than a tenth of a degree.

Without weights, the answer depends on the altitude. With the low body at ten degrees the four-sight fix lands 1.11 kilometres from the ship on average, better than the 1.21 of the three high sights alone. At five degrees it lands 1.38 kilometres off and at three degrees 1.72: the low sight has made the fix worse than not taking it. The crossing is at 7.2 degrees. Below it, a navigator who averages every line equally would do better to throw the low one away.

With weights, there is no crossing. The weighted fit with the low body at three degrees lands 1.16 kilometres off — better than the three high sights, better than the unweighted fit at every altitude — and at ten degrees 1.06. This is not a lucky property of the layout. Weighting each line by the reciprocal of its variance gives the best linear unbiased estimate, the Gauss–Markov theorem in the form Alexander Aitken gave it in 1935, and a best estimate cannot be improved by throwing away data: at worst a useless sight is weighted to nearly nothing and the fix is the fix without it. A weighted sight is never worse than none. An unweighted one can be much worse.

The old rule of thumb is that sights below about ten degrees are best avoided, and the measurement says what that rule actually is. It is not a statement that low sights are useless. It is the correct response to low sights in an unweighted fit, which is what a navigator plotting lines of position with a pencil is doing, and it is the wrong response to them in a weighted one.

The altitude depends on the weather, not on the stars #

The 7.2 degrees belongs to a tenth of the refraction correction being uncertain. Change the share and the crossing moves.

At a twentieth uncertain the crossing is three degrees and a low sight is nearly always worth taking. At a fifth it is fourteen, at a half twenty-eight. The layout matters less than might be expected: the body behind three high ones and the body at one of four quarters cross at almost the same altitude, and the low body sharing a side with the high ones crosses about a degree higher, because its line adds less geometry to the fix and so there is less to set against its noise. The weather is the larger term by far. A ten-degree rule corresponds to about fifteen per cent of the correction being uncertain, a reasonable guess for ordinary conditions at sea and a poor one for a flat calm over cold water.

That makes the rule of thumb a statement about the atmosphere disguised as a statement about the sky. The number a navigator actually needs is not an altitude but a share, and the share is the one quantity nobody on deck can measure.

Weighting turns the region, and the unweighted region points the wrong way twice #

A weight changes not only where the fix lands but the shape of the region it is likely to be in, and the confidence ellipse is where that shows.

The weighted fit’s ellipse is its own inverse normal matrix, built with each line counted by its weight. The low body behind the ship is the line that constrains the fix along the north–south direction the high bodies share, and a weighted fit trusts it less, so the ellipse stretches along that direction and turns towards it. An unweighted navigator draws something else: the inverse of the matrix with every line counted once, scaled by an average variance. And the unweighted fix’s real scatter is a third shape, because the estimator that trusted every line equally is spread by the low line’s large error in a direction neither of the other two ellipses describes.

With the low body at five degrees, the drawn unweighted ellipse is turned twenty degrees from the weighted one, and twenty-eight from the scatter of the very fix it is drawn around. It describes neither the best available fix nor the fix it belongs to. Above twenty degrees the three ellipses come together within a few degrees, which is the same altitude at which the sights’ errors come together.

This is the question a confidence ellipse is honest only where the sheet keeps angles ended on: whether weights turn the region enough to change which shoal is inside it. The answer is yes, and not in the direction the question expects.

The weighted ellipse, with the low body at five degrees, lies wholly inside the ellipse the unweighted navigator draws. A shoal inside the weighted region is inside the drawn one too, so a navigator who weights loses no warning that one who does not would have had. The danger is on the other side. The unweighted fix does not scatter inside the ellipse drawn for it: twelve per cent of the region it really falls in lies outside that ellipse, in two lobes along its long axis. Along that axis the drawn ellipse ends 2.45 kilometres from the fix and the fix’s real 95 per cent region runs on to 2.98. A shoal two and a half to three kilometres off in that direction is where an unweighted fix can put the ship, and outside the region the navigator drew to say where the ship might be.

The unweighted ellipse keeps its coverage by being the wrong size #

A region drawn in the wrong shape might be expected to hold the ship less often than it claims. Measured exactly, it barely does, and the reason is worth having because it would otherwise read as reassurance.

The unweighted fix with its ellipse drawn from the average of the four variances holds the ship 92.9 per cent of the time with the low body at three degrees, against a nominal 95. That looks almost honest. It is honest the expensive way: the ellipse is more than twice the area of the weighted one, and its extra area is spread round the whole circumference rather than along the direction the fix is actually uncertain in. A navigator drawing it is paying for coverage in paper, and still falls short. The answer is a set argues for quoting a region rather than a point; the region is only as good as the variances it is drawn from, and an unweighted fit has thrown the variances away before drawing it.

The more realistic unweighted navigator does not compute an average variance at all. The sextant’s accuracy is the number a navigator knows, and an ellipse drawn from it is an ellipse drawn as if refraction were exact. That region holds the ship 78 per cent of the time with the low body at five degrees and 65 per cent at three. It is the right shape for nothing and the wrong size for everything, and it is the defect the residual reports the error the fix was immune to warned about from the other side: a statement of accuracy computed from the sights’ agreement with a model that left out the one error that matters.

The weights are a guess, and the guess has a safe direction #

Everything above assumed the navigator knows the share of the refraction correction that is uncertain. Nobody does. The weights are a guess the solve believes made the same point about a survey network, where weights nobody measured moved coordinates by a factor of 1.8; at sea the guess is about the air, and it can be wrong in either direction.

The two directions do not cost the same. Guess the share at half its true value and the fix lands 1.20 kilometres off instead of 1.13, and its ellipse holds the ship 90 per cent of the time instead of 95. Guess it at twice the true value and the fix lands 1.16 off and its ellipse holds the ship 96 per cent of the time. Guessing half is worth about twice the accuracy loss of guessing double, and it loses coverage where guessing double gains it.

Push both further and the asymmetry grows. A guess of a quarter gives 83 per cent coverage and 1.31 kilometres; a guess of four times gives 98 per cent and 1.20. At the extremes the two directions arrive at two familiar fits. Guessing the refraction exact is the unweighted fit, 1.38 kilometres and a region that holds the ship 78 per cent of the time. Guessing it hopeless weights the low sight to nearly nothing and arrives within a hundredth of a kilometre of the three high sights alone, 1.22 against 1.21, with an ellipse so large it holds the ship every time and says nothing.

So a navigator who has to guess should guess high. An over-cautious weight degrades the fix gently towards the one without the low sight, which is a good fix, and its ellipse errs on the side of holding the ship. An over-confident weight degrades the fix towards the unweighted one, which below seven degrees is worse than both, and its ellipse errs on the side of losing the ship. The error in the guess is not symmetric because the two fits it falls back to are not.

How the numbers were checked #

Every rate and distance above is computed from the linear algebra of the fit rather than from trials, so the algebra itself needs checking against the thing it summarises.

The fits must coincide when nothing distinguishes the sights. With all four bodies at forty-four degrees, every sight has the same variance, and the weighted fit, the unweighted fit and the unweighted navigator’s ellipse must be the same matrix. They agree to 10⁻¹².

The weighted fit must never lose. At every altitude from two degrees to forty-five, its root-mean-square distance from the ship must be no larger than either the unweighted fit’s or the three-sight fit’s. It is not, at any of the thirteen altitudes tested, which is what the Gauss–Markov theorem requires and what a mistake in the weights would have broken.

The coverage integral must reproduce a case with a closed form. For an ellipse drawn at twice the true variance of a circular error, the share inside is exactly $1 - e^{-c}$ with $c = 5.99$, and the integral gives it to within 2×10⁻⁴. The weighted fit’s own ellipse must hold the ship 95 per cent of the time to the same tolerance, and does at every altitude.

With no refraction uncertainty, no sight may be worse than none. At a share of zero every sight has the sextant’s error, and adding one can only help. No crossing altitude exists.

The algebra must agree with sights. Four thousand seeded trials run through the whole reduction — bodies placed at their altitudes and azimuths, each sight’s error drawn from its own variance, lines reduced from an assumed position twenty kilometres off and plotted on Mercator — with the low body at four degrees give a weighted fit 1.158 kilometres from the ship against the algebra’s 1.148, an unweighted one 1.521 against 1.515, and a weighted ellipse holding the ship 94.8 per cent of the time.

What the weights leave out #

The errors are independent. A common error — an unknown dip of the horizon, an index error — shifts every line together and was the subject of what the extra unknown costs where nothing can see it. An abnormal refraction near the horizon is partly common too, since every low body is seen through the same layer, and two low bodies would share part of their error in a way this model does not represent.

The share is one number for every body. The uncertainty of refraction is not really a fixed share of the correction at all altitudes. Close to the horizon anomalous refraction is generally held to grow faster than the correction itself, in which case the curves here are kind to the lowest sights.

The ellipses are of the estimate, and an error ellipse is an indicatrix carries through unchanged: on a sheet that does not keep angles the weighted ellipse is displaced and misread exactly as the unweighted one was, and weighting does nothing to repair a sheet.

Still open: a sight that is weighted by what it disagrees with #

The weights here come from outside the sights: from an altitude and a stated share of a correction. A fifth or sixth sight offers another source. Once there are more lines than unknowns, the residuals say something about which lines disagree with the rest, and a fit can re-weight each line by its own residual and solve again — the iteratively reweighted least squares that robust estimation uses to push down an outlier.

Whether that helps a navigator is not obvious, and the reason is the one the residual reports the error the fix was immune to found. A residual sees only the part of an error the geometry does not absorb. A low body standing alone behind the ship carries the most geometry of any line, so its error is absorbed into the fix and its residual is small precisely because the fit depends on it. A scheme that trusts lines with small residuals would trust that one most. How much of a low sight’s error its own residual can reveal, at which layouts residual-based weights converge on the altitude-based ones and at which they converge on the opposite, is a question weights stated in advance cannot ask.