Precision — the ladder · Cartographic Projection

What the 5th decimal place of a coordinate is worth on the ground. A latitude written to 5 decimal places steps 1.112 m north for one unit in its last digit, at every latitude, because the meridian does not care where it is measured. A longitude written to the same 5 places steps 1.112 m east on the equator and 0.097 m at 85°, because a degree of longitude is a degree of a circle whose radius is R cos φ. The same written precision means two different distances at the same point, and a different pair at every other.

A coordinate is a number with a width

Every number on this site so far has been exact. A written coordinate is not: five decimal places of a degree is 1.112 metres of latitude everywhere and 1.112 metres of longitude only on the equator, falling to 0.097 at 85°. The same written precision carves the ground into a cell that is square in one place and eleven times longer than it is wide in another.

rung 1 · distortion

One instrument, one covariance, drawn where Mollweide puts it. The same measurement is made at every point: 5 m east by 5 m north, uncorrelated, which on the ground is a circle. Each ellipse is that covariance pushed through the projection's own Jacobian and drawn 26,000 times life size. On Mollweide the axis ratio reaches 4.15, so an instrument that is equally good in every direction is drawn as though it were not.

An error ellipse is an indicatrix

A positional covariance pushed through a projection is the same matrix sandwich that produces Tissot's indicatrix, so the error ellipse drawn on a map and the distortion ellipse drawn beside it are the same ellipse. A five-metre circular accuracy is drawn at an axis ratio of 3.04 on one common projection — and the same projection draws a genuinely lopsided 304 by 100 metre error as a perfect circle.

rung 2 · distortion

10,000 observations of one place, averaged 120 times. Left: 400 single observations of the same place, with a 60-kilometre standard deviation east and north, drawn across ±400 km. They are scattered about the truth and their average is unbiased on the ground. Right: 120 independent averages of 10,000 such observations each, computed on the page and taken back to the ground, drawn across ±1.8 km. The cloud is tight, as averaging ten thousand things should make it, and it is not centred on the cross: it sits 489 m away, against 403 m predicted by the projection's second derivative alone.

The average of noisy positions moves

Average sixty thousand scattered observations of one place on a Mercator map and the answer is 404 metres too far north — at every sample size, because it is a bias and not noise. The same average on the Lambert cylindrical equal-area is 404 metres too far south, the two being ½ (σ²/R) tan φ and its exact negative, and on the plate carrée it is not displaced at all.

rung 3 · distortion

A chain of stations, its positions' uncertainty and its baselines'. Nine stations held at the left-hand one, with every neighbour, second neighbour and third neighbour observed. The filled ellipses are each station's own uncertainty, which grows without limit as the chain runs away from the point it is held at — 7.5 mm at the near end and 574 mm at the far. The open ellipses above each leg are the uncertainty of the baseline, drawn at the same scale: they hardly grow at all, because almost everything that is wrong with one end is wrong with the other in the same direction.

The difference of two coordinates

Three essays give a single coordinate a width. Every practical use of one is a difference of two — a distance, a bearing, a movement, an area — and the width of a difference is not the two widths combined, because the errors are not independent. Far from its datum a one-leg baseline is six times more certain than the positions it joins.

rung 4 · distortion

A 900 km circular accuracy at 55° north, projected. Six thousand ground positions drawn from a circular error of 900 kilometres about one place, each projected in Mercator and plotted as a displacement from the projected place. The curve is the nominal 95 per cent ellipse, computed the standard way — the ground covariance sandwiched between the projection's own derivatives. It holds 93.83 per cent of the points, the cloud is measurably longer than it along its own long axis by 5.52 per cent, and it is not symmetric: the third moment along the page's second axis is 0.727 rather than zero.

The error ellipse is not an ellipse

Rung two pushed a covariance through a projection with the same matrix sandwich that draws an indicatrix. That is a first-order operation on a map with a second derivative, so the propagated distribution is not the ellipse the sandwich draws — and a nominal 95 per cent ellipse holds 93.06 per cent on one projection and 95.63 on another, in opposite directions, from the same input.

rung 5 · distortion

Waiting longer buys almost nothing under the spectrum that is actually there. The scatter of block means against the block length, for the three spectra. White noise falls with a slope of -0.484 — the inverse square root everybody assumes. Flicker falls with a slope of -0.111, so a hundredfold longer average buys a factor of 1.67 rather than ten. A random walk is very nearly flat.

The error that does not average down

Five rungs give a coordinate a width, and all five assume the observations it was averaged from are independent. They are not. Under the noise spectrum every published analysis of a position time series reports, the standard error falls as the eleventh root rather than the square root — so a factor of ten costs a hundred observations if the noise is white and a thousand million if it is not.

rung 6 · distortion

The bias is σ² over the length, over two decades. The amount by which a measured baseline is longer than the true one, against its length, for a twenty-millimetre error on each end. The line is the second-order prediction σ²/d. The measured bias times the length is constant to a factor of 1.0014 across the whole range, and it sits 8.2 per cent below the prediction — which is the fourth-order term the expansion drops. The estimate is antithetic, so the first-order scatter cancels exactly and a bias of thirty-seven microns is measured at a t-statistic of 349.

A length measured from noisy points is too long

A distance is a square root, a square root is concave, and the average of the distances is not the distance between the averages. The gap is a bias with one sign: 37 microns on a ten-metre baseline with twenty-millimetre marks, following σ² over the length across two decades, and it adds rather than cancelling — so the same boundary is 1.5 parts per million longer when it is measured in more pieces.

rung 7 · distortion

Densifying helps one and hurts the other. The standard deviation of the shoelace area, and the bias of the perimeter, against the number of vertices on the same 100-metre circle with the same 5-centimetre noise on each. The area's spread falls from 9.9 m² to 2.0 — the closed form says it goes as the square root of the vertex count's reciprocal, because a vertex's influence on the area is the vector between its two neighbours and densifying shortens it. The perimeter's bias rises from 0.07 mm to 261, a factor of 3697, because every leg contributes its own σ²/d and shorter legs contribute more.

The area is unbiased and the perimeter is not

A boundary measured from noisy vertices comes out long, always, by σ²/d on every leg. The area enclosed by the same vertices comes out exactly right, because a shoelace is bilinear and the cross terms vanish. So densifying a boundary makes its area five times more precise and its perimeter three thousand times more wrong, and every compactness score computed from it falls short.

rung 8 · distortion

Two places, one written coordinate. The lattice a coordinate written to 5 decimal places lives on, at 45° of latitude, where a cell is 1.11 metres north to south. Two real places a fifth of a cell apart are drawn as open marks and the single value they both round to as a filled one. The distance between them, computed from what was written, is exactly zero — an error a noise model cannot produce, since independent noise never puts two different points in the same place.

Rounding is not noise

Eight rungs treat a coordinate's error as noise that averages down. A published coordinate has a second error that does not: it is deterministic, it is shared between every point in the same cell, and at five centimetres apart ninety-three per cent of pairs come out as the same place — which no amount of independent noise can produce.

rung 9 · distortion