The error ellipse is not an ellipse

The second rung of this ladder found a genuine identity: a positional covariance pushed through a projection is the same matrix sandwich AΣAᵀ that produces Tissot’s indicatrix, so the error ellipse drawn on a map and the distortion ellipse drawn beside it are the same ellipse. A five-metre circular accuracy is drawn at an axis ratio of 3.04 on one common projection.

That result is exact and it is first order. A is the projection’s first derivative, the sandwich is what a linear map does to a Gaussian, and a linear map sends a Gaussian to a Gaussian. A projection is not linear. So the sandwich answers a question about a linearised projection, and the ladder has never asked at what accuracy the linearisation stops being the map.

What the sandwich is entitled to say #

Write the projection as f, the ground position as x, its covariance as Σ. Expand about the true place:

$$f(x) = f(\mu) + A(x - \mu) + \tfrac{1}{2}(x - \mu)^{\mathsf T} H (x - \mu) + \dots$$

The sandwich keeps the first two terms. If the third were absent the propagated distribution would be exactly Gaussian with covariance AΣAᵀ, and every statement anyone makes about an error ellipse would be exactly right.

The third term is not absent, and this ladder has already met it once. The average of noisy positions moves is the displacement of the mean: E[f(x)] − f(μ) is ½ tr(HΣ), which is 404 metres on a Mercator map for a 60-kilometre scatter, at every sample size, because it is a bias rather than noise.

That rung took the mean. This one takes the shape, which the same term also changes and in three ways at once: the spread grows, the ellipse turns, and the distribution stops being symmetric about its own centre.

The sandwich’s own output is worth having in front of the argument, because none of what follows says it is wrong. It says it is incomplete, and the two are different criticisms: at a five-metre accuracy every number in that figure is right to more decimal places than anyone can use.

Measuring it without measuring the sampler #

There is a trap here that had to be closed before any number below could be trusted.

The obvious experiment is: draw forty thousand ground points with covariance Σ, project them, take the sample covariance of the results, and compare with AΣAᵀ. That comparison measures two things — the projection’s nonlinearity, and the fact that a finite sample does not have exactly the covariance it was drawn from. At forty thousand points the second is 0.7 per cent in the axis ratio, which is the same size as the effect being looked for at ordinary accuracies.

So the comparison here is against AΣ̂Aᵀ, where Σ̂ is the sample covariance of the very same ground points. The sample’s own luck then appears identically on both sides and cancels exactly. What is left is the projection, and it is the reason the numbers in the tables below start at four decimal places of agreement rather than at two.

What the ellipse actually holds #

The measurement a reader cares about is not the covariance, it is the containment: a curve labelled 95 per cent is a promise about how often the truth is inside it. Measured on Mercator at 55° north:

At a surveying accuracy the promise is kept to two decimal places. At a thousand kilometres the ellipse holds 93.06 per cent, which is nearly two points short — one truth in fourteen falls outside a curve that was drawn to let one in twenty out.

And the sign is the projection’s to choose #

The interesting part is that the error is not always in that direction. At a thousand kilometres:

Three of the eight hold more than they promise. An ellipse that over-holds is exactly as wrong as one that under-holds and is far less likely to be reported, because nothing fails: the truth is inside the curve more often than advertised, every consistency check passes, and the region is simply larger than the data justifies.

Mollweide and Albers are the two equal-area members, and they are the two that over-hold most. That is not a coincidence — an equal-area map’s second-order term compresses where the first-order term stretches, so the tail of the propagated cloud is pulled in rather than pushed out — but it is not a rule either: the Lambert cylindrical is also exactly equal-area and it under-holds.

What a reader does with the ellipse #

An error ellipse is not a decoration; it is used for three things, and the second-order failure costs each of them differently.

A containment claim. The true position is inside this curve with 95 per cent probability. That is the reading the table above prices directly, and the cost is between −1.94 and +0.63 percentage points at a thousand-kilometre accuracy.

A comparison. These two ellipses overlap, so the two fixes may be the same place. This one is worse, because two ellipses computed at different latitudes have different second-order errors, and an overlap test compares two curves that are each wrong by a different amount in a different direction. Nothing in an overlap test can see that.

A weight. This observation is worth 1/σ² in the adjustment. Here the second-order term does not merely change the region, it changes the weight, and a weight that is systematically too small at one end of a network and too large at the other is exactly the failure the reduction ladder found the residuals cannot see — because a residual tests the model it was computed under.

The three are ordered by how visible the failure is, and it is the reverse of how much it costs.

Where the effect arrives #

The flexion ladder’s seventh rung asks at what size a second derivative arrives, and the same question has an answer here. Taking half a percentage point of lost containment as the threshold, at 55° north:

Five hundred kilometres of circular accuracy sounds absurd for a position and is not: it is the right order for a historical position reconstructed from a description, a low-frequency direction-finding fix, an animal recovered from a ring return, a seismic epicentre from a sparse network, or a satellite footprint quoted as a nominal centre. Every one of those is routinely plotted with an ellipse on a small-scale map, which is the only kind of map on which a five-hundred-kilometre ellipse fits.

And the same measurement at the equator moves Mercator’s arrival from 492 km to 1,188 — the effect is a property of the place as much as of the projection.

Why it stays hidden until it does not #

The excess spread grows as the square of the accuracy, at a fitted exponent of 1.96 to 2.01 across three projections. The second-order term is quadratic in the displacement, so this is the law it must obey, and measuring it is the check that what is being seen is that term rather than an artefact.

A square law is the shape that hides. Over the range anybody tests at — metres to a few kilometres, where the effect is parts in ten million — it is indistinguishable from zero, and confidence in the sandwich is built there. Then it arrives quickly: from 0.02 per cent at 100 kilometres to 8.67 per cent at a thousand, a factor of four hundred for a factor of ten in the input.

That is the same shape as every other second-order finding on this ladder and on the flexion ladder next door, and it is why both ladders exist. A first-order model is not wrong; it is right with an error that is quadratic, and quadratic errors are invisible in the regime where the model is validated.

Three things change, not one #

It is worth separating what the second-order term does, because the ellipse is wrong covers three different failures.

The spread grows. On Mercator at 55°, the cloud’s long axis exceeds the sandwich’s by 0.25 per cent at 200 km, 1.10 at 400, 4.02 at 700 and 8.67 at 1,000.

The ellipse turns. The measured principal axis is 7.6° from the predicted one at 200 km and 55° at 1,000. On a conformal projection this number needs care: the first-order ellipse there is a circle, and a circle has no orientation, so what the measurement records is that the propagated cloud has an orientation the sandwich cannot supply at all. That is a stronger statement than a turn.

The distribution stops being symmetric. The third moment along the page’s second axis rises from 0.011 at a 10 km accuracy to 0.846 at 1,000 km. A Gaussian has a third moment of zero and an ellipse is the contour of a Gaussian; a distribution with a skewness near one is not one, whatever ellipse is drawn round it.

The three are not independent — all three come from the same H — but they fail different checks. The first is caught by comparing spreads, the second by comparing orientations, and the third by nothing anybody runs, because a covariance has no third moment in it and there is nowhere for the number to be reported.

The place where there is nothing to find #

Everything above is a departure from a prediction, and a departure is only evidence if the measurement is capable of returning zero.

Mercator’s ground Hessian — the second derivative of the page coordinates with respect to a displacement east and north in metres — is 6.1 × 10⁻¹⁴ at 55° north and 2.5 × 10⁻²³ at the equator, which is zero to the arithmetic’s own precision. Every second derivative of its page coordinates with respect to ground displacement carries a factor that is stationary on the equator.

So at the equator the propagated distribution really is an ellipse, and the measurement says so: the containment stays within 0.19 of a percentage point of 95 at every accuracy up to 1,500 kilometres, against 3.05 points at 55° on the same projection with the same sampler and the same code. The skewness stays at a constant 0.0043 — the sampler’s own residual — rather than rising with σ.

That is the refusal this rung needs. The effect vanishes exactly where the term responsible for it vanishes, on the same projection, so what is measured everywhere else is the second derivative and not the experiment.

A note on which covariance the ladder means #

Rung four separated the width of one coordinate from the width of a difference of two, and the same care is needed here about what Σ is.

Everything above takes Σ as an accuracy on the ground, in metres east and north — which is what an instrument produces, what a datum realisation quotes, and what a published coordinate’s stated uncertainty is a statement about. Pushing it onto the page is then a modelling step with the failure this rung measures.

The other direction — a covariance quoted in page units, in a projected coordinate system, which is how most files store it — has the same failure mirrored. A page ellipse pulled back to the ground through the inverse projection is a first-order object too, and its ground meaning has the same quadratic error. There is no direction in which the sandwich is exact, because the sandwich is a linearisation and the map is not linear either way round.

What does change between the two is where the error is discovered. A ground-quoted accuracy is compared against ground measurements and the discrepancy shows up in the field; a page-quoted one is compared against other page quantities, all carrying the same linearisation, and it agrees with itself indefinitely.

What to do about it #

Three responses, in increasing order of effort, and the first is nearly always enough.

Say where the ellipse was computed. An error ellipse propagated on the page is a page object; propagated on the ground it is a ground object; and the two differ by everything above. Almost no published ellipse says which it is, and saying so costs a clause.

Draw the contour rather than the ellipse. The 95 per cent contour of the propagated distribution is computable — it is what the cloud in the hero has — and it is not an ellipse, so drawing it requires a polygon rather than two axes and an angle. That is a change to a data format, which is why it does not happen.

Or propagate on the ground and project the contour. The ground covariance is what the instrument measured and the ground is where the uncertainty lives. Push the contour through the projection rather than the matrix, and every number above goes away, because the projection is then applied to a curve rather than to a linearisation.

Who actually has a thousand-kilometre uncertainty #

The regime where the effect arrives sounds hypothetical to a surveyor, and it is worth naming the fields that live in it, because they are the ones that will meet this without any geodetic tradition to warn them.

Coarse geolocation. A position inferred from an IP address, from a cell-tower fingerprint at low density, or from a radio direction-finding fix has an uncertainty of tens to hundreds of kilometres and is routinely represented as an ellipse on a web map.

Light-level geolocation in animal tracking. A tag that records daylight and infers position from sunrise and sunset times gives latitude uncertainties of one to two hundred kilometres, worse near the equinoxes, and the published figures are ellipses drawn on projected sheets.

Fall and re-entry footprints. A meteorite strewn field or a re-entering satellite’s predicted footprint is an uncertainty region hundreds to thousands of kilometres long, and it is drawn as an ellipse because that is what the propagation produced.

Historical and archaeological provenance. A find placed by a description rather than by a survey carries an uncertainty of the size of a region, and it is increasingly stored as a covariance so that it can be handled by the same tools as everything else.

None of those fields has a reason to distrust the library. The propagation is standard, the code is the same code, and the ellipse is drawn without complaint. What differs is only the scale of the input, and the scale of the input is the whole of the effect — which is why the failure is waiting in exactly the places that have no history of thinking about projections at all.

The remedy is the one aboveThe list is also a reminder that the regime is growing rather than shrinking, since every one of those fields has moved from describing a position in words to storing it as a covariance.

The remedy is the one above and it is no harder in these cases than in any other: propagate on the ground and project the contour. What is missing is the knowledge that the shortcut has a range of validity, since nothing in the tool says so.

What this rung establishes #

The propagated error ellipse is a first-order object and the projection is not. The departure is second order in the accuracy, at a fitted exponent of 1.96 to 2.01, so it is invisible at survey accuracies and arrives between 480 and 1,148 kilometres on four of eight common projections.

A nominal 95 per cent ellipse holds between 93.06 and 95.63 per cent at a thousand-kilometre accuracy, depending on the projection, and the sign of the error is the projection’s to choose. Over-holding is the more dangerous of the two because nothing downstream fails.

And the effect is exactly the second derivative, confirmed by the place where that derivative vanishes and the effect vanishes with it, on the same projection, at 2.5 × 10⁻²³ against 6.1 × 10⁻¹⁴.

The ladder’s second rung said the error ellipse and the indicatrix are the same ellipse. They are — and what this rung adds is that neither of them is the shape of the thing once the uncertainty is large enough to draw at the scale the map is printed at.