A coordinate is a number with a width

One hundred and fifty essays on this site have computed something exactly. A projection’s four partial derivatives are differentiated to eleven figures, a datum shift is seven parameters and a rotation matrix, a scale factor is quoted to four decimal places because four decimal places fit on the axis. Nothing has ever been said about how wide any of those numbers is.

The subject does not work that way. A coordinate arrives written down, with a finite number of digits, and those digits are a claim about a patch of ground rather than about a point. The patch has a size and a shape, both are computable from the projection the coordinate belongs to, and neither of them is what the usual rule of thumb says.

The rule of thumb, and where it is true #

The most repeated version is six decimal places is about eleven centimetres. It appears in software documentation, in data specifications, and in the answer to every question about how many digits to keep.

It is right, on the equator, for a longitude, on a sphere. Everywhere else it is one of two other numbers.

At six decimal places the meridian step is 0.1112 m, at every latitude, and that number is genuinely constant — on a sphere exactly and on the ellipsoid to within the one per cent the meridian radius of curvature varies by. The parallel step is 0.1112 m at the equator, 0.0556 m at 60° and 0.0097 m at 85°. Nothing about the coordinate says which of the two applies, because both do: a coordinate has two numbers and they are quantised differently.

The spread between ellipsoids is the part of this usually discussed, and it is the smaller half: a degree of latitude differs by more between the equator and the pole on one ellipsoid than it does between three ellipsoids at one latitude. The quantisation question lives entirely in the larger half.

The rule of thumb is not wrong in the way a mistaken formula is wrong. It is a measurement of one case, repeated as though it were a property of the notation.

What a written coordinate actually stands for #

Rounding both numbers of a geographic coordinate to five decimal places says: the true position is somewhere in a rectangle 1.112 m tall and 1.112 cos φ m wide. That rectangle is the ground the coordinate cannot distinguish, and its shape is a fact about where it is.

Three quantities describe the cell and they do not move together:

• its area, which falls as cos φ — 1.24 m² at the equator, 0.62 at 60°, 0.11 at 85°;
• its aspect ratio, which is 1/cos φ — 1.00, 2.00 and 11.47 at the same three latitudes;
• its worst error, which is half the diagonal, and which barely moves at all: 0.786 m at the equator and 0.558 m at 85°.

The third is the one anybody quotes and the one that says least. A coordinate written to five places is good to about three quarters of a metre everywhere, and that summary conceals the fact that at 85° the uncertainty is almost entirely north–south. Two positions ten centimetres apart east–west get different coordinates; two positions eighty centimetres apart north–south may not.

The grid coordinate is a different quantisation of the same ground #

A projected coordinate rounded to the metre carves the ground into a square in grid units, which is not a square on the ground. What it becomes is the indicatrix acting on a square, which is the same construction as an area on the grid not being an area on the ground arriving from the other direction.

On a conformal grid the shape survives and the size does not:

Both columns are exactly what the projection’s own name promises, and both are startling when read as precision rather than as distortion. A metre of Mercator grid at 80° north resolves seventeen centimetres of ground in both directions — the coordinate is six times more precise than its unit suggests, and no field in its definition says so. A metre of an equal-area grid at 80° resolves a sliver 5.75 m long and 0.17 m wide, with the same area as a square metre and none of its usefulness.

The pattern is worth stating plainly, because it is the ladder’s first general result. A geographic coordinate quantises the ground into a cell whose shape is decided by latitude; a grid coordinate quantises it into a cell whose shape is decided by the projection. The first is a property of the parameterisation and the second is a property of the map, which is the distinction what survives a change of coordinates is built on, met again in a place it was not expected.

Two quantisations that do not agree #

Software that stores a geographic coordinate and draws it on a pyramid of tiles is quantising twice, and the two lattices are not aligned.

East-west the two lattices stay in a fixed ratio, because both scale as $\cos\varphi$. North-south they do not, because the stored step is constant and the drawn one is not — so the two cross over at a latitude, and that latitude is where a stored coordinate stops being finer than the picture it is drawn on.

Below the crossover the stored precision is finer than the pixel and the rounding is invisible. Above it the stored coordinate is the coarser of the two, and a feature drawn from it snaps to a lattice a reader can see. Neither the storage format nor the display says which regime a given map is in; the latitude does.

The digits nobody chose #

A coordinate held in a 32-bit float has a precision nobody wrote down anywhere, and it is not a property of the coordinate system either.

The two curves coinciding is the finding. A float carries 24 bits of mantissa, so the spacing between representable values is about one part in sixteen million of the value itself. Degrees and grid metres both span roughly 20,000 kilometres from origin to antipode, so both reach the same relative magnitude at the same place on the ground, and both resolve about two metres there. Changing the unit changes nothing.

What does change it is the origin. At 400 km from the origin the same float resolves 0.031 m; at 20,000 km it resolves 2.00 m, a factor of 64. A false easting is chosen so that coordinates in the region of interest come out positive and have a convenient number of digits — an origin that is not there — and choosing it also chooses, silently, how finely single-precision arithmetic can resolve the ground. A grid with a false easting of 400,000 m resolves three centimetres near its central meridian. The same grid extended to a coordinate of 20,000,000 m resolves two metres.

In double precision the same calculation gives 3.7 nanometres and the question disappears, which is why it is now rare and why it was once ubiquitous.

A tolerance written in degrees is not a circle #

The same arithmetic runs one step further out. A tolerance — within 0.01 degrees, snap points closer than 0.0001 — is a distance written in the units of the coordinate, and in geographic coordinates it is not a distance at all.

Every deduplication threshold, snapping distance and coincidence test written in degrees is an ellipse of this shape, oriented north–south, growing more eccentric towards the poles. Two points a kilometre apart north–south at 75° are inside a 0.01° tolerance; two points three hundred metres apart east–west are outside it. The rule is stated as though it were about distance and it is about the coordinate.

What was computed, and how #

Every number here comes from two sources and neither is a citation.

The ground step of a written digit is R cos φ Δλ and R Δφ with Δ = 10⁻ᵈ degrees, evaluated at a mean radius of 6,371.0088 km. The check on it is the equator, where the two must be equal, and the figure asserts that they agree to nine figures there and disagree by more than tenfold at 85°.

The quantisation cell is built by taking the four corners of the rounding rectangle — in degrees for a geographic coordinate, in grid units for a projected one — carrying them through the projection’s own linear map at the point, and measuring the polygon that comes back with the shoelace formula. That map is localFrame: the two columns are the page displacement produced by one metre of ground east and one metre north, and its singular values are Tissot’s semi-axes. The grid cell’s ground shape is therefore not a separate calculation from the distortion; it is the distortion, applied to a square.

The assertion the figure carries has to fail as well as pass. It requires the geographic cell to be square on the equator, not square at 80°, square on the ground for a conformal grid at any latitude, and not square for an equal-area one — four claims, of which two are refusals. A version that only checked the first two would pass on a broken projection library.

The floating-point curve computes the spacing directly from the exponent, 2^(e−23), rather than by differencing adjacent floats, so that it can be evaluated at magnitudes no coordinate system uses and the shape of the curve is visible rather than inferred.

The digits people actually write #

The number of decimal places a coordinate arrives with is almost never chosen. It is whatever the format’s default was, whatever the previous system emitted, or whatever fitted the column — and because the choice is invisible it is never revisited when the data is reused for something the original precision cannot support.

Two habits follow from the arithmetic above and are worth stating as habits rather than as findings. Quote the precision as a ground distance, not as a digit count, because the digit count means two different things in the same coordinate and a different pair of things at every latitude. And write the same number of ground metres rather than the same number of places, which for a geographic coordinate means one more decimal place on the longitude near the poles than on the latitude — an asymmetry no format supports and every dataset would benefit from.

Neither is likely to be adopted. What can be adopted is the observation underneath them: a coordinate’s written precision is a claim about a patch of ground whose shape is computable, and computing it takes one line.

Where the model stops #

Three limits, all of them stated rather than skirted.

The cell of a rounded geographic coordinate is a rectangle in latitude and longitude, and its image on the ground is treated here as a parallelogram — the linear map at the centre applied to the corners. At five decimal places that is exact to far more figures than anything else here; at four decimal places, over eleven metres, the curvature of the parallel contributes about a part in 10¹². It would matter for a coordinate rounded to whole degrees, which nothing does.

The single-precision curve is about spacing, not about accumulated error. A calculation done in single precision does not merely quantise its answer, it loses digits at every step, and a chain of transformations can be far worse than its output’s spacing suggests. That is a question about arithmetic rather than about coordinates and it is not asked here.

And rounding is not the only way a coordinate is written. Degrees, minutes and seconds quantise the ground differently again — one second of arc is 30.9 m of latitude, so a coordinate written to whole seconds is coarser than five decimal places by a factor of 27 — and the sexagesimal and decimal lattices do not nest. The comparison is arithmetic and it is left as an observation rather than a figure.

The two lattices a coordinate can be written on #

The note above about degrees, minutes and seconds is worth finishing, because the interaction between the two notations is where a stored coordinate quietly loses its provenance.

A coordinate in whole seconds sits on a lattice of 1/3600 of a degree, which is 30.87 metres of latitude — coarser than five decimal places by a factor of 27.8, and coarser than four by a factor of 2.8. A coordinate in tenths of a second sits on a lattice of 3.087 metres. Neither lattice contains any point of any decimal one, because 1/3600 has a factor of 3² in its denominator and every decimal step has only twos and fives, so a value in whole seconds has no terminating decimal expansion at all.

That has a consequence a data pipeline meets and rarely names. Converting a coordinate from seconds to decimal degrees is a rounding, always, and how much it costs depends on how many places the destination carries. Four decimal places has a step of 11.1 metres against the second’s 30.9, so distinct whole seconds still land on distinct decimals and the conversion is reversible — barely, at a ratio of 2.8. Three decimal places has a step of 111 metres and is not: several distinct seconds collapse onto one decimal value, and the original can no longer be recovered.

The reversibility threshold is the useful number, and it is the one the arithmetic gives directly: a decimal step finer than half the source lattice preserves every distinct source value. Whole seconds need four decimal places, tenths of a second need five, hundredths need six. Anything beyond that is recording the rounding rather than the measurement.

What no format records is which lattice the coordinate started on. A value of 51.4833° could be a measurement good to a hundred metres, or 51° 29′ 00″ converted, or a five-decimal value with its last digit dropped, and the three carry different widths and different shapes of width. The digits are the same in all three cases, which is the whole difficulty of the ladder this rung opens: a written number carries its precision only by convention, and the convention is not written down anywhere in the file.

The generalisation #

The result that survives leaving cartography is the one the ladder is named for.

A written number is not a value, it is an interval, and applying a nonlinear map to it maps the interval as well as the value. Every field that writes coordinates down meets this: a timestamp to the second is an interval, an instrument reading to three figures is an interval, and the moment either passes through a function that is not linear, the interval’s shape changes and its width stops being a property of the notation.

What cartography adds is that the map doing the changing is one whose derivatives are already tabulated. The distortion machinery this collection is built on — the Jacobian, the two principal scales, the areal factor — is exactly what turns a written precision into a patch of ground, with no new apparatus at all.

Who found it, and when #

The arithmetic is elementary and old, and its consequences have been rediscovered by every generation of software.

The rule that a degree of longitude shortens as the cosine is in the earliest tables of the dead reckoning navigator; the ratio was called the departure and the correction for it was known as plane sailing, which is why a degree is not a unit of length has such an old vocabulary attached to it. The specific problem of choosing how many decimal places to write is entirely modern, and dates from the moment coordinates began to be exchanged as text rather than as marks on a sheet.

The floating-point half is younger still. The IEEE 754 standard fixed the single-precision mantissa at 24 bits in 1985, and the two-metre quantisation that follows from it has been rediscovered in map software approximately once per decade since — usually as a report that features “jitter” when the map is panned a long way from the origin, which is exactly what a coordinate lattice that coarsens with magnitude looks like when it is drawn.

Where the ladder goes next #

This rung takes the width of a coordinate from the way it is written. The next takes it from the instrument that produced it, which is a covariance rather than a rounding interval — and finds that pushing one through a projection is not a new calculation at all, but the figure this site drew first.