A confidence ellipse is honest only where the sheet keeps angles

Three essays here have measured celestial fixes and none of them has drawn a region the ship is likely to be in. A cocked hat holds the ship one time in four established that the triangle three lines bound is not such a region — it holds the ship a quarter of the time, whatever its shape — and the two essays after it worked entirely in point estimates and their mean distances from the truth.

The region a fix does have is the confidence ellipse of its covariance, and it is the same object an error ellipse is an indicatrix draws for a surveyed coordinate. It arrives at a navigator’s plot with one extra hazard that a surveyed coordinate does not have: the plot is drawn on a sheet the navigator chose, and every essay behind this one ends by measuring what an uncorrected sheet does.

Seven square kilometres of confidence, and fifty of measurement #

The two pictures come from the same three sights and the same arithmetic. The hat is where the lines cross each other; the ellipse is where the estimate might be. They have almost nothing in common, and their sizes are the clearest statement of that.

The cocked hat covers seven square kilometres and holds the ship a quarter of the time. The ellipse covers fifty and holds it ninety-five per cent — which is neither an achievement nor a coincidence, because ninety-five per cent is what it was constructed to hold and the measurement is a check that the construction is right rather than a discovery.

What that comparison buys is the exchange rate. Two regions can only be compared at equal confidence, and once they are, a sight’s worth becomes readable: adding a fourth body takes the ellipse from fifty square kilometres to forty, a fifth of the paper, at the same ninety-five per cent. The cocked hat cannot be put on that scale at all, because its confidence is a quarter and no navigator wants a quarter. To use a hat as a region at ninety-five per cent one would have to inflate it by a factor the geometry decides afresh for every set of azimuths, and having done so it would not be a triangle any more.

What a hat would have to be inflated by is worth stating, because it shows that the two pictures are not a scale apart but a kind apart. A hat of the average size at the surrounding layout is seven square kilometres and holds the ship a quarter of the time; the ninety-five per cent ellipse is fifty. The factor is about seven on area at that layout — and it is about seven at the one-sided layout too, where the hat is 6.89 and the ellipse 50.2, which looks like a rule until one notices that the hat’s own size varies with the azimuths in a way that has nothing to do with how well the ship is fixed. The hat is small when the three lines happen to cross near a point, and a cocked hat holds the ship one time in four measured directly that a small hat is not a good fix: hats in the smallest third of a sample and hats in the largest third have centres the same distance from the ship. An inflation factor applied to a quantity that does not measure accuracy cannot produce one that does.

There is a reason the older picture persists, and it is not ignorance. The hat is drawn with a pencil and a straight edge from things already on the sheet. The ellipse needs a two-by-two matrix inverted and a chi-square percentile, which is a minute’s work with a table and was not a minute anybody had on a wet bridge wing. The answer is a set makes the general case for quoting a region rather than a point; the navigator’s version is older than the argument and had a better excuse.

Where the two numbers come from #

The ellipse is not an extra construction bolted onto the fix. It is the same matrix the fit already inverted, read differently. Each line of position contributes its unit azimuth vector to a two-by-two normal matrix; the fit inverts that matrix to find the point, and the inverse, scaled by the per-sight variance, is the covariance. Its eigenvectors are the ellipse’s axes and its eigenvalues their lengths, and the ninety-five per cent contour is that shape scaled by the square root of 5.99, the chi-square percentile on two degrees of freedom.

So the ellipse costs nothing extra to compute once the fix has been computed, and it is available at three sights as well as four. It is the same object that decides, in the residual reports the error the fix was immune to, how much of a common error a two-unknown fit can absorb: the share that survives into the residual is a statement about which directions the normal matrix can reach, and the ellipse is a picture of how far it reaches in each. One matrix, three readings — the point, the region, and what the region cannot see.

The shape is where the geometry shows #

An ellipse has two numbers and a direction where a mean distance has one. The arc the bodies span moves them very unevenly.

From a forty-degree bunch to the four quarters the ellipse’s area falls by a factor of 2.03 and its axis ratio by 3.79. The short axis hardly moves: 2.53 kilometres at forty degrees and 3.46 at two hundred and seventy — it is longer in the good geometry, because a circle of the same area has to be. Everything the arrangement of the bodies does to a fix, it does to the long axis and the direction it points in.

That has a practical edge. A navigator who quotes one number for a fix’s quality — a radius, a probable error — is quoting something close to the mean of the two axes, and the mean is the part that barely moves. The thing that distinguishes four bodies in one quarter of the sky from four spread round it is entirely in the ratio and the bearing, and those are exactly what a single number discards. The same complaint is made of a projection’s distortion by the average was a choice of norm: a scalar summary of a two-dimensional quantity chooses, silently, which of its two dimensions to care about.

What the sheet does to the centre #

Now the sheet, and the first of two failures. On a plotting sheet with no correction for latitude, every line of position carries the first-order error the intercept method’s own accuracy measured, and the ship is plotted somewhere it is not.

On Mercator and on a corrected plotting sheet the ellipse holds the ship 95.1 per cent of the time at every assumed-position error out to forty kilometres. On the uncorrected sheet at fifty degrees north it holds it 92.3 per cent of the time at two kilometres of assumed-position error, 70.2 at five, 12.1 at ten and not once in three thousand trials at twenty.

The collapse is entirely in the centre. The ellipse is the right size and the right shape on all three sheets — the lines of position are laid off identically on every one of them, so the covariance in sheet coordinates does not depend on the sheet at all. What differs is where the sheet says the ship is, and by ten kilometres of assumed-position error that disagreement has exceeded the ellipse’s own long axis.

This is the most dangerous failure a quoted region can have, and it is worse than a region that is too small. A region too small is caught by experience: a navigator who quotes ninety-five per cent and is outside half the time finds out. A region of the right size in the wrong place looks correct on every plot, is the right size when compared against other fixes, and is wrong at a rate the navigator has no way to sample — because being wrong requires knowing where the ship actually was.

It is the same shape of defect what the extra unknown costs where nothing can see it found for a common sextant error at a bunched layout: an answer that is wrong in a way the answer’s own diagnostics are constructed not to notice. There the diagnostics were the residual; here they are the drawn region’s size and shape. Both are statements about the internal agreement of the sights, and a displacement that moves everything together leaves internal agreement untouched.

Swept over latitude at a fixed assumed-position error of twenty kilometres, the uncorrected sheet gives 95.1 per cent at ten degrees, 90.9 at twenty, 62.9 at thirty, 6.8 at forty and nothing from fifty north. The transition is fast — most of it happens between twenty-five and forty-five degrees — because the first-order error grows as one less the cosine of the latitude while the ellipse’s own size does not grow at all, so the ratio of the two crosses from negligible to fatal across a narrow band.

That band is where a great deal of the North Atlantic’s traffic has always sailed. The plate carrée, the projection nobody chooses is the map that happens when nobody decides on one, and a plotting sheet laid down without its latitude correction is that map at the scale of a single fix: not chosen, not noticed, and exactly wrong in the latitudes where it is used.

What the sheet does to the shape, which is a different fault #

The second failure has nothing to do with the assumed position and does not cost a single point of coverage. It is about what the drawn ellipse means.

A sheet coordinate measured east–west does not stand for the same number of kilometres on the ground as one measured north–south, so reading the drawn ellipse back through the sheet’s own graticule gives a different region from the one the sights implied. At fifty degrees north an ellipse of 4.02 by 3.08 kilometres becomes one of 3.75 by 2.13, turned twenty-six degrees.

This is invisible on the plot. The lines, the fix, the ellipse and the ship’s plotted position are all drawn in the same frame and all consistent with one another; nothing looks odd. The discrepancy appears only when somebody converts the drawing back to latitudes and longitudes — reporting a position and its accuracy, handing the fix to another vessel, entering it in a log that another sheet will read.

The area ratio is the cosine of the latitude at every latitude tested, to nine decimal places — which is the reciprocal of the plate carrée’s own areal factor there. The misreading is not an extra error introduced by the fix or by the ellipse. It is the sheet’s distortion, applied once, to whatever is drawn on it.

That identification is the point rather than a remark, and it is the same matrix sandwich an error ellipse is an indicatrix uses to show that an error ellipse pushed through a projection and the distortion ellipse drawn beside it are the same ellipse. Here the sandwich arrives from the navigator’s end: what a sheet does to an uncertainty is what it does to a small circle, because an uncertainty is a small circle as far as the sheet is concerned. And the error ellipse is not an ellipse is the correction to both, since the transformation is first-order on a map with a second derivative — negligible here, at four kilometres, and not negligible at the thousand-kilometre accuracies that measurement was about.

Why the two failures do not mix #

It is worth being explicit that these two are independent, because a reader who has followed both might reasonably expect them to compound.

The centre error grows with the assumed position’s own error and vanishes when the navigator’s estimate is good. It is the navigator’s, in the sense that better dead reckoning removes it. The shape error depends on the latitude alone, is the same for a perfect assumed position as for a hopeless one, and no care in observing or reckoning touches it.

They do not mix because they live in different places in the same arithmetic. The first moves the ellipse’s centre relative to the ship; the second changes the linear map from sheet coordinates to ground coordinates. A coverage test done in sheet coordinates and one done in ground coordinates give identical answers, because the map between them is linear and invertible — which is exactly why the shape error costs no coverage, and exactly why it costs meaning instead.

So the account of the plotting sheet is now complete in a way no single measurement made it. The sheet displaces the fix, which two of the essays behind this one measured. It displaces the region, which is the same fault seen larger. And it misdescribes the region’s shape, which is a separate fault, is not detectable from the plot, and is the projection’s own distortion doing to an uncertainty exactly what it does to everything else drawn there.

What each number was checked against #

A perfect assumed position must clear every sheet. With the assumed position exactly on the ship, all three sheets plot the ship at one point, so the coverage must be identical to the last bit. It is: the three shares differ by less than 10⁻¹². A difference there would have meant the test itself was sheet-dependent, and every other number in this essay would have been unreadable.

The nominal rate must be nominal where the sheet keeps angles. On Mercator and on the corrected plotting sheet, the ellipse must hold the ship within two points of ninety-five per cent at every assumed-position error tested from nothing to forty kilometres, and does.

And must fail where it does not. On the uncorrected sheet at fifty degrees north with the assumed position twenty kilometres out, the coverage must fall below one in twenty, and falls to nothing in three thousand trials.

The shape error must be the sheet’s own areal factor, not something like it. The ratio of the denoted area to the true one must equal the cosine of the latitude to within 10⁻⁹ at every latitude from the equator to eighty degrees, and does. An approximate agreement would have meant the effect was partly something else.

And the turning must be real. An oblique ellipse at fifty degrees north must be turned by more than twenty degrees, and is turned by 25.7. The control sits beside it: an ellipse aligned with the meridian is turned by nothing at any latitude, because a diagonal scaling cannot rotate its own eigenvectors, so a figure drawn with a symmetric layout would have shown no turning and proved nothing.

The ellipse must beat the hat, and a fourth sight must shrink it. From the same three sights the ellipse must hold the ship above nine times in ten where the hat holds it below three, and a fourth sight must reduce the ellipse’s area; both hold at both layouts.

What a region drawn on a sheet leaves out #

The errors are Gaussian and known. A confidence ellipse is a statement about a normal distribution with a stated variance. Sextant errors have tails that are not normal — a misidentified star is a different kind of event — and a navigator estimating the variance from four sights has one degree of freedom to do it with.

Ninety-five per cent is a choice. Nothing here argues for it. A navigator standing off a lee shore wants a region that holds the ship far more often than nineteen times in twenty, and the ellipse scales by the square root of a different chi-square percentile without any of the arithmetic changing.

The ellipse is the region of the estimate, not of the ship. The two coincide because the estimator is unbiased, which is true of the two-unknown fit only when no common error is present. With a common error the ellipse is centred on the biased point and its coverage falls exactly as the sheet’s does — a third displacement of the centre, with a different cause and the same effect.

The sheet is one of three, and the three are not exhaustive. Mercator, a corrected plotting sheet and an uncorrected one are the sheets a navigator plots on. A gnomonic chart, used for laying off great circles, keeps no angles at all and would fail differently again — its distortion is not a diagonal scaling, so its version of the shape error would not be a simple cosine.

And the second failure was measured on one oblique layout. The area ratio is the cosine of the latitude for any ellipse whatever, since it is a determinant; the turning angle depends on the ellipse’s own orientation and runs from nothing to a maximum that this essay did not chart.

Still open: the region when the errors are not all of one kind #

Every region here is drawn from one variance shared by every sight. A real set of sights is not like that: a body forty degrees up is observed through less atmosphere than one at eight degrees, its refraction correction is smaller and better known, and the two sights deserve different weights. Weighting changes the covariance and so changes the ellipse’s shape and bearing, not merely its size.

That opens a question equal weights cannot ask. Two bodies of quite different altitude give a fix whose ellipse is not the one an unweighted fit draws, and whether the difference is worth a navigator’s attention — whether the ellipse’s bearing moves by enough to change which shoal is inside it — depends on how unequal real sights are. The weights themselves are a guess, as the weights are a guess the solve believes has had occasion to say, and a region drawn from guessed weights has a confidence that is itself uncertain by an amount nobody quotes.