The residual reports the error the fix was immune to

Three sights and three lines make a triangle, and a cocked hat holds the ship one time in four measured what that triangle can and cannot be read as. It ended on a warning that has no instrument behind it: a common error in every altitude — a sextant with an index error, an unallowed height of eye — moves all three lines together, and on one side of the sky it announces itself by swelling the triangle while on the other it quietly carries the ship out of it.

A line of position is Newton’s method, but only on a conformal chart established that the construction itself adds almost nothing on a conformal sheet, so everything that follows is the sights’ own error and the sheet’s. Nothing in three sights can tell a navigator which of the two has happened. Three lines of position and three unknowns — two coordinates and the common error — is a square system: the point equidistant from three lines always exists, always fits exactly, and never leaves anything over to be suspicious of.

A fourth sight breaks the square. Four lines over-determine two coordinates by two and three unknowns by one, and for the first time in this account of a celestial fix the arithmetic has something left in its hands when it has finished.

What a fourth sight is worth, and where #

Before the leftover, the fix. Each line of position is one linear equation $\mathbf{n}\cdot\mathbf{x}=c$, with $\mathbf{n}$ the unit vector towards the body and $c$ set by the intercept. Four of them are an over-determined system, and three conditions are one too many meets the same surplus from the other direction, where a third exact condition on a two-parameter construction has no freedom left to be satisfied with. Here the surplus is welcome: the standard answer is the point minimising the sum of squared distances to all four — the same estimator a coordinate is the output of a solve describes at the other end of this subject, where a surveyed network has hundreds of equations for dozens of unknowns.

Over six thousand simulated sets of sights, each body observed with an independent error of two kilometres along its azimuth, the fourth sight is worth about eighteen per cent of the distance from the ship when the bodies stand at the quarters of the compass, and thirty-three per cent when they are bunched within sixty degrees of each other. That ordering is not the obvious one. The bunched sky gives the worse fix at every count — 3.64 kilometres against 2.18 at three sights — and it is also the sky in which one more sight helps most, because the fourth body adds a direction the other three were nearly blind in, and at the quarters there is no such direction left to add.

The share of fixes landing within three kilometres of the ship tells the same story from the other side: 89 per cent at the quarters and 71 bunched. A navigator choosing which fourth star to shoot is choosing where to spend a sight, and the arithmetic says to spend it where the sky is thin.

The mechanism is visible in what a fix is made of. Two lines crossing at a fine angle fix the position well along one direction and badly along the one perpendicular to it, which is the elongated uncertainty an error ellipse is an indicatrix draws as an ellipse rather than a circle. Bodies bunched in one quarter of the sky give four nearly parallel lines, so three of them repeat one another’s information and the ellipse is long; a fourth body anywhere outside the bunch shortens the long axis, which is where nearly all of the error is. At the quarters the ellipse is already nearly circular after three, and the fourth sight can only shorten what is already short. The general form of the rule is that an observation is worth what it adds in the direction the others are weakest in, which is the same principle by which where the control points are decides what a fit can recover before any of it is measured.

Two unknowns leave two degrees of freedom, and they are a measurement #

What the fourth sight adds that three could not is the leftover itself. With four lines and two unknowns, the fitted point sits at four distances from four lines, and those four residuals cannot all be made zero.

Their size is a statement about the sights. Under independent errors of standard deviation $\sigma$, the sum of squared residuals divided by $\sigma^2$ has a chi-square distribution on two degrees of freedom, so its expectation is exactly two, and the root-mean-square residual is a direct estimate of how well the sights agree with each other. Averaged over three thousand sets, it is 1.27 kilometres for sights good to two, against the 1.41 the distribution gives as the root mean square of that quantity — the gap between them being the ordinary one between the mean of a square root and the square root of a mean.

This is the quantity the weights are a guess the solve believes calls the variance of unit weight, and it is the first honest self-check a celestial fix has had available to it here. A three-sight fix quotes a precision that is entirely an assumption; a four-sight fix quotes one it has tested. The answer is a set makes the same demand of an identification: a best fit with no spread beside it is not a measurement, and until the fourth sight there was no spread available to quote.

The two quantities move in opposite directions as sights are added, which is worth a moment because it looks wrong. The fix gets better — 2.16 kilometres at three sights, 1.46 at six, 1.03 at twelve — while the residual gets larger, from 0.90 to 1.78. They are measuring different things. The fix’s distance from the ship falls because averaging more observations sharpens an estimate. The residual rises because with three lines and two unknowns only one degree of freedom is left, and a single leftover systematically understates the scatter in the way a sample of one always does; by twelve lines the root-mean-square residual has climbed to within about a tenth of the true two kilometres.

So more sights do not merely make a better fix. They make a more trustworthy one, and the trust arrives on a separate channel.

That separation is worth stating plainly, because it is what everything below turns on. The fix is a statement about where the ship is. The residual is a statement about whether the sights are consistent with each other. They are computed from the same four numbers and they are not two views of one quantity: a set of sights can be beautifully consistent and badly wrong, or scattered and, on average, right. The rest of this essay is about the first of those, which is the one a navigator cannot detect by looking at the plot.

The share of a common error that survives #

Now the question three sights could not ask. A common error of $b$ kilometres adds the same $b$ to every one of the four right-hand sides. Does it show?

The answer is a piece of linear algebra with no probability in it. Fitting two coordinates projects the four observations onto the two-dimensional space the azimuths span. The part of a common error the fit can absorb by moving the position is the part of the all-ones vector lying in that space; the part it cannot absorb is what reaches the residual. Writing $\mathbf{A}$ for the matrix of unit azimuth vectors and $\mathbf{P}$ for the projection onto its column space, the surviving share is

$$\kappa = \frac{\lVert(\mathbf{I}-\mathbf{P})\mathbf{1}\rVert^{2}}{m}$$

It is the redundancy of one particular direction, and it is the same construction the blunder the network cannot see uses for a single observation, where the redundancy number of a leg decides what share of a gross error in it reaches the residuals and what share is absorbed into the coordinates. The difference is which direction is being asked about: a network asks about one observation at a time, and a navigator suspecting a sextant has to ask about all four at once.

Bodies at the four quarters of the compass give $\kappa=1$ exactly, because the four unit vectors sum to zero and the all-ones direction is orthogonal to everything the fit can reach. Every kilometre of a common error passes into the residual untouched — and, by the same orthogonality, the fix does not move at all. Four bodies bunched within sixty degrees give $\kappa=0.0041$: the fit absorbs 99.6 per cent of a common error by sliding the position sideways, and four-tenths of one per cent arrives at the residual.

A half-turn gives exactly one half, which is the same landmark the cocked hat’s containment reached by a quite different argument — there it was the angle at which the point equidistant from three lines changes from being inside their triangle to being outside it. The two calculations have almost nothing in common. One counts which side of each line the ship falls on; the other is a projection in a four-dimensional space of observations. That they meet at a half-turn says the quantity being divided is the same one: whether the bodies’ directions surround the ship or lie within a half-plane, which decides whether a step towards every body at once is a step the position can make.

The arithmetic at the two ends is short enough to check by hand. At the quarters, the four unit vectors are the four compass quarters and sum to zero, so $\mathbf{A}^{\mathsf{T}}\mathbf{1}=\mathbf{0}$ and no combination of a northward and an eastward shift can imitate a step towards every body. Bunched at 60°, 80°, 100° and 120°, the four vectors sum to a vector of length 3.70 pointing very nearly north, and $\mathbf{A}^{\mathsf{T}}\mathbf{A}$ has 3.44 in its northward diagonal: moving the position 1.08 kilometres south imitates a one-kilometre common error in all four sights to within four parts in a thousand.

The instrument is sharpest where nothing needs measuring #

Put the two halves together and the result is the disagreeable one.

At the quarters, a common error of eight kilometres takes the residual from 1.27 to 8.01, which no navigator could miss. But at the quarters a common error moves the fix by nothing: the least-squares point is 1.79 kilometres from the ship with no common error and 1.79 with four kilometres of one, to the third figure. The instrument screams about a fault that is doing no harm.

Bunched within sixty degrees, a common error of four kilometres moves the fix from 2.44 kilometres from the ship to 4.01, and takes the residual from 1.29 to 1.30. The instrument is silent about the fault that is doing all the harm.

This is not a defect of the least-squares estimator or of the residual. It is one conservation statement seen twice: a common error is either absorbed into the position or left in the residual, and there is nowhere else for it to go. The sky that hides it from the residual is hiding it inside the answer, which is the only other place available.

What a test on the residual actually catches #

A navigator does not read a residual as a number so much as compare it with what the sights ought to give. The formal version is a test: reject whenever the sum of squared residuals exceeds the ninety-fifth percentile of its no-error distribution, which fires falsely one time in twenty by construction.

At the quarters the test is excellent: 76 per cent power against a three-kilometre error and 95 against four. On one side of the sky over a hundred and twenty degrees it reaches 17 per cent at four kilometres. Bunched within sixty degrees it reaches 5.2 per cent, against the 4.7 it fires at with nothing wrong at all — the false-alarm rate itself, and the honest description of that is not a weak test but no test.

What a closed figure cannot see makes the same point about a surveyed traverse: a misclosure checks only the errors a closure is sensitive to, and a systematic error along the traverse’s own direction closes perfectly. Four sights bunched in one quarter of the sky are a closure with the same blind spot, and the blind direction is towards the bodies, which is precisely the direction a sextant error moves every line in.

The reciprocal, and what it costs to ask #

There is an obvious response to all this: if four lines over-determine two unknowns by two, solve for the common error as a third unknown and stop guessing. The system still has a degree of freedom left over.

It works, and it is priced by the same $\kappa$. Solving for the common error inflates the position’s variance by a factor whose value, over every layout tested, is exactly $1/\kappa$ — 1.00 at the quarters, 2.00 at a half-turn, 11.7 on one side over a hundred and twenty degrees, and 243.8 bunched within sixty. Where the residual can see a common error, removing it is free. Where the residual is blind to it, removing it costs a factor of two hundred and forty in variance, which is a fix scattered fourteen kilometres from the ship in place of one scattered two and a half.

It is worth being clear that this is not a coincidence of the four layouts chosen. The product of the surviving share and the variance inflation is 1 to within $10^{-6}$ for all five stated layouts and for arbitrary ones — five bodies at 0°, 37°, 111°, 264° and 300°, three at 5°, 25° and 45°. The two quantities are the same piece of geometry written twice, once as what is left outside the fit’s reach and once as what it costs to reach for it.

What that exchange is worth is a separate calculation, and stating it here only sets the price. What matters now is that the two questions — is it visible? and can it be removed? — are not two questions. They are one number read in two directions, and it is settled by the choice of bodies, before a sextant leaves its case.

The sheet the residual cannot see at all #

One more blindness, and it belongs to this subject rather than to statistics. Both of the essays behind this one end on a plotting sheet whose longitude is not corrected for latitude, where every line of position carries a first-order error that grows with the assumed position’s own error.

The fix’s distance from the ship on the uncorrected sheet grows to 6.35 kilometres at an assumed position twenty kilometres out, and to 24.97 at eighty. The residual, over the same range, is 1.276 kilometres on both sheets — the same to three figures at twenty kilometres, and 1.311 against 1.311 at eighty.

The reason is the same conservation argument in a third costume. The sheet’s error shifts every line by nearly the same amount in nearly the same direction, so the four lines go on agreeing with one another perfectly while agreeing with the Earth less and less. Consistency is a relation among the observations; correctness is a relation between the observations and the ground, and a residual only ever measures the first. A residual has more than one explanation makes this the general rule for fitting a projection to a map; here it has its sharpest special case, because the residual is not merely ambiguous about the chart — it is exactly, measurably constant in it.

The two blindnesses are not quite the same, which is worth separating. A common error is partly visible, at a share the geometry fixes, and there are skies in which it is fully visible. A wrong sheet is invisible at every geometry tested, because it does not act along the azimuths at all: it displaces the whole plot, lines and ship together, and a displacement of everything leaves every distance between the lines untouched. No choice of bodies improves it, and no number of sights does either — twelve lines on the uncorrected sheet agree with one another as perfectly as four.

So the fourth sight buys a real instrument and buys it with a stated blind spot: it sees disagreement among sights, and it cannot see anything all four sights share. A wrong sheet is shared — and the plate carrée, the projection nobody chooses is the map that happens when nobody decides on one, which at the scale of a single plot is exactly the sheet measured above. A sextant’s index error is shared. The two faults a navigator most wants a warning about are the two the warning is made deaf to.

What each number was checked against #

Three lines and three unknowns must leave nothing over. Fitted with the common error as a third unknown, three lines give a residual below $10^{-9}$ kilometres over zero degrees of freedom, whatever errors are put in. A construction that left a residual there would be making its own.

The quarters must give exactly one, and exactly one. For bodies at the four quarters, $\kappa$ and the variance inflation must both be 1 to nine decimal places, and are: the four unit vectors sum to zero.

The reciprocal must be exact, not approximate. For all five stated layouts and for two arbitrary ones — five bodies at 0°, 37°, 111°, 264° and 300°, and three at 5°, 25° and 45° — the product of $\kappa$ and the inflation is 1 to within $10^{-6}$.

A narrow bunch must hide a common error twice over. Four bodies within sixty degrees must give $\kappa$ below 0.01 and an inflation above 100, and give 0.0041 and 243.8.

The residual must move at the quarters and not at the bunch. A three-kilometre common error must more than double the residual at the quarters and must leave a narrow bunch’s within a tenth of where it was; it moves 1.27 to 3.13 and 1.29 to 1.29.

The two-unknown fix must be exactly immune at the quarters. A four-kilometre common error must move the least-squares point there by under two per cent of its own scatter, and moves it by under one.

And the test must be the size it claims. Every detection curve must start at one in twenty with no error present, and they start at 0.056, 0.048 and 0.047 over four thousand trials each.

What four sights and a known ship leave out #

A common error is the only shared error modelled. Two bodies observed low in the same haze share part of their refraction, which is neither independent nor common to all four; the whole apparatus here rests on a fault having exactly the all-ones shape, and $\kappa$ would have to be recomputed for any other.

The sights are simultaneous. Four stars observed over a quarter of an hour from a ship making fifteen knots must be run up to a common time, and an error in the run shifts the lines along the ship’s track rather than along their own azimuths — a third shape again.

The errors are stated rather than observed. Two kilometres of independent scatter is an ordinary sextant at sea, and the shape of every curve here is fixed by $\kappa$ rather than by that number; the level of every residual is proportional to it.

And the test assumes the scatter is known. Comparing the residual against a chi-square percentile uses a $\sigma$ the navigator has taken on trust. A navigator who estimates $\sigma$ from the sights themselves is testing a residual against a number computed from the same residual, which is a weaker test than the one measured here and a different calculation.

Still open: what the fifth unknown costs when it is not needed #

The reciprocal above prices removing a common error and says nothing about whether to. Solving for it is free at the quarters and ruinous at a bunch, but that is only the variance; a navigator has to decide with four numbers in hand rather than a distribution, and the decision is not symmetric. Ignoring an error that is there leaves a bias; solving for one that is not there costs scatter, and at a narrow bunch the scatter costs more than any plausible sextant error would have.

Nothing here says where the crossing is. Whether the estimated common error can be trusted at all when $\kappa$ is small — its own standard deviation at a narrow bunch is larger than the errors anybody is looking for — whether the residual can be used to choose between the two fits without invalidating both, and whether there is a rule better than either that shrinks towards zero by an amount the geometry sets, are questions the price alone does not answer.