A degree is not a unit of length

A specification says within 0.01 degrees. A query asks for everything inside a box a tenth of a degree across. A gridded dataset is published at half-degree resolution. All three treat a degree as though it were a distance, and none of them says which distance.

The two radii #

The ground distance a degree spans is not one number because the ellipsoid has two radii of curvature at every point.

$$\text{north–south: } M(\varphi)\frac{\pi}{180}, \qquad \text{east–west: } N(\varphi)\cos\varphi\,\frac{\pi}{180}$$

$M$ is the meridian radius and $N$ the transverse one, and they are the same quantities the curvature of the Earth is not one number uses to show that the Gaussian curvature varies by 1.35 per cent from equator to pole. Here they do something simpler and more consequential: they make a buffer written in degrees an ellipse.

Measured for a hundredth of a degree:

The equator, where a sphere would say one #

The first row is the one worth stopping at. On a sphere the ratio at the equator is exactly one — a degree is a degree in both directions — and here it is 0.9933, so the shape is 6,694 parts per million shorter north–south than east–west.

That is the flattening arriving where nobody expects it. At the equator $N = a$ and $M = a(1 - e^2)$, so the meridian’s radius of curvature is smaller than the parallel’s by a factor of $1-e^2$, and the ellipse’s minor axis is the north–south one. The Earth is flattened at the poles, so the meridian at the equator is the most sharply turning part of it — which is the same counter-intuitive fact found in the other direction elsewhere here, when the first version of an assertion said the pole was the most curved place and the monotonicity check refused it.

The site’s gate asserts this rather than the obvious thing: the ratio at the equator must differ from one by more than $10^{-3}$, which fails if the calculation has quietly reverted to a sphere and would pass if it only checked that high latitudes give tall ellipses.

And 60° is not exactly 2:1 either #

The rule of thumb is that the ratio is $\sec\varphi$, which at 60° is exactly 2. The measurement gives 1.9966, short by 1,684 parts per million, because the true ratio is

$$\frac{M(\varphi)}{N(\varphi)\cos\varphi}$$

and $M/N$ is below one everywhere except the pole. The gate checks the measured ratio against that expression to twelve decimal places and against 2 not at all, which is the difference between verifying a derivation and confirming a memory.

Sixteen hundred parts per million is 1.6 metres per kilometre. Nothing about a query tolerance cares. It is stated because the alternative — printing $\sec\varphi$ and calling it measured — is the practice this whole site exists to replace. The correction is $M/N = (1-e^2)/(1-e^2\sin^2\varphi)$, which is the flattening entering in the same place, and by the same route, as it does in geodetic against geocentric latitude — where it reaches 11 arcminutes rather than a part in a thousand.

The north-south half of the same story is smaller and not zero. A degree of latitude runs from 110.574 kilometres at the equator to 111.694 at the pole — a variation of one per cent, which is why the vertical axis of a degree buffer looks constant and is not. Three ellipsoids in common use differ from each other by less than that variation within any one of them, which is the useful way round: the choice of ellipsoid matters less here than the choice of latitude.

The one latitude where the ellipse is a circle #

The first two rows of the table straddle something the essay has not yet said out loud. At the equator the ratio is 0.9933, so the shape is wider east–west than north–south; at 15° it is 1.0288, so it is taller than it is wide. Between them it passes through one, and there the degree buffer is an exact circle on the ground.

Solving $M(\varphi) = N(\varphi)\cos\varphi$ puts it at 6.5898° of latitude, where a hundredth of a degree is 1,105.89 metres in every direction. That is the only latitude on the ellipsoid at which within 0.01 degrees is a statement about a distance rather than about a coordinate — a circle of a stated radius, isotropic, with no direction in it.

Two things about that number are worth pausing on. It is not zero, which is the answer a spherical treatment gives and which is wrong by 6.6°: on a sphere the ratio is $\sec\varphi$, monotone, equal to one only at the equator. And it is not a property of the Earth’s size — it is fixed by the eccentricity alone, since the condition reduces to $(1-e^2) = \cos\varphi\,(1 - e^2\sin^2\varphi)$, in which the semi-major axis cancels. Every ellipsoid with WGS84’s flattening has its circle at the same latitude whatever its radius, and an ellipsoid with a different flattening has it somewhere else.

So the family of ground shapes a degree buffer makes runs from a flattened ellipse at the equator, through a circle at 6.59°, to increasingly tall ellipses everywhere above — and the passage through a circle is the reason the equatorial row of the table is so easily read as a rounding error. A ratio of 0.9933 looks like a number that ought to be one. It is a number that is one, six and a half degrees further north.

The practical form is small and worth having. Anyone quoting the sec φ rule is quoting a rule that is exact nowhere and worst near the equator, where it says one and the truth is 0.9933, and again at the circle latitude, where it says 1.0066 and the truth is exactly one. The rule’s own error is largest precisely where the quantity it approximates is closest to trivial, which is why nobody has ever noticed it: the places where sec φ is badly wrong are the places where the answer is nearly one either way.

The condition a specification actually wants #

Almost every use of a degree tolerance means a distance. Written the other way round — a thousand metres, converted to degrees — the numbers are:

so the east–west figure nearly quadruples over the range while the north–south one moves by less than one per cent. Any single degree value chosen to mean “a kilometre” is therefore a kilometre in one direction at one latitude and something else everywhere else.

What the ellipse does to a count #

A tolerance that encloses a different amount of ground at every latitude produces counts that are not comparable, and the arithmetic is the same as the buffer’s.

A study that counts occurrences within 0.01° of each of a set of points is counting within 3.87 square kilometres at the equator and 1.95 at 60° north — the areas the table’s last column gives as 99.6 and 50.3 per cent of nominal. Any rate computed from those counts inherits the factor of two, and the factor is a function of latitude alone, so it is perfectly correlated with anything else that varies north to south.

That is the failure mode worth flagging: not noise, but a systematic gradient aligned with latitude, produced entirely by the coordinate system and indistinguishable in the output from a real north–south trend. The repair is the same one as always and costs one conversion; the reason it is worth naming is that the result of not making it looks like a finding.

Why it persists #

Not because anybody believes a degree is a distance. Because the alternative costs something, and the something is worth naming precisely.

A geodesic buffer — every point at a stated ground distance — is the right answer and is not a shape with a simple description in the coordinates the data is stored in. Its boundary has to be computed point by point through an inverse geodesic problem, which is Vincenty’s iteration or an equivalent, and geodesics on the ellipsoid sets out why there is no closed form.

A projected buffer — project to something local, take a circle, project back — is exact if the projection’s scale is true over the region, and choosing for a line, not a region shows how much the choice of projection matters when the objective is a distance rather than an area.

A degree buffer costs two additions. That is the whole of its appeal, and it explains the persistence completely: it is the only one of the three that can be done without knowing anything about the Earth.

What it does to a query #

A tolerance that is an ellipse produces answers that depend on direction, and the effect has a shape.

At 60° north a “one hundredth of a degree” query returns everything within 1,114 metres north or south and 558 metres east or west. Two candidate points at 800 metres, one due north and one due east, are treated differently: the northern one is inside and the eastern one is outside. Nothing in the data distinguishes them; the coordinate system does.

That is the same failure nearest is a question about the metric measures for a nearest-neighbour query, and the two belong together: a threshold and a ranking are the same operation with different outputs, and both inherit whatever metric the plane they were computed in happens to have.

What it does to a grid #

Gridded data is usually published on a fixed-degree grid, and a cell of a fixed number of degrees is a cell of varying area:

computed from the closed form $R^2\Delta\lambda(\sin\varphi_2 - \sin\varphi_1)$, which needs no dataset and has no error bar — the same closed form this site chose over a coastline shapefile at foundation, for reasons the projection that shows true size sets out.

The consequence is that an unweighted average over such a grid is not an average over the Earth. A mean taken cell by cell counts a polar cell as heavily as an equatorial one four times its size, which biases the answer towards high latitudes by exactly the ratio of the cell areas. The repair is to weight by $\cos\varphi$, which is the areal element and takes one multiplication; the failure is invisible in the arithmetic and shows up only against a differently-gridded source.

The rule of thumb, priced #

“A degree is 111 kilometres” is the version of all this that people carry, and it is worth pricing rather than dismissing, because it is better than it has any right to be in one direction and useless in the other.

North–south it is right to within one per cent everywhere: the true value runs 110.574 to 111.694 kilometres, so the rule is 0.39 per cent high at the equator and 0.62 per cent low at the pole, and it is exact at 38.19°. For a rough distance that is an excellent rule.

East–west it is right at the equator, 13 per cent low at 30°, 41 per cent low at 45°, and wrong by a factor of four at 75°. As a rule about a degree rather than about a degree of latitude, it is therefore correct along one direction and one line, which is the same shape of failure as every other quantity in this field.

The honest compression is two rules rather than one: a degree of latitude is 111 kilometres everywhere; a degree of longitude is 111 kilometres times the cosine. That costs one more word and one more operation, and it removes the entire east–west column of the error.

A box query is the same failure with corners #

The buffer’s ellipse has a rectangular cousin: the bounding box, which is how most spatial queries are actually written.

A box of $\pm0.1°$ about a point at 60° north spans 22.3 kilometres north–south and 11.2 east–west, so it is not a square of ground, and its north edge is shorter than its south edge because the parallels converge. On the ground it is a trapezoid, narrower at the top by 0.3 per cent over this box and by more over a larger one.

Its area follows the closed form rather than the product of its sides: $R^2\Delta\lambda(\sin\varphi_2 - \sin\varphi_1)$, which is exact and which differs from width times height by the same convergence. For queries that is a curiosity. For anything that divides by the area — a density, a rate, a count per square kilometre — it is the whole answer, and the next essay in this ladder is about nothing else.

The same cell drawn on four projections becomes four different shapes of four different areas, all from one piece of ground — which is the reason a “degree square” is a description of a coordinate box and never of a piece of the Earth.

Three quantities that are not lengths #

It is worth collecting the family, because a degree is the most common member and not the only one.

A degree, which is an angle, and whose ground length depends on latitude and direction: everything above.

A pixel, which is a cell of a projected plane, and whose ground size depends on the zoom and the latitude: the pixel is a place with a size.

A grid unit, which is a metre on the projection and not a metre on the ground: the line scale factor, which the practice field integrates along a line rather than sampling because the difference reaches metres.

All three are units of something, and the something is a coordinate rather than a distance. The confusion is always the same one and it is always available, because the coordinate is what a file contains and the distance is what a question is about.

The one place a degree really is a length #

There is an exception, and naming it keeps the essay honest: along a meridian, a degree is a distance to within one per cent everywhere on Earth, and the one per cent is the flattening rather than the projection.

That is not a small exception. It is why nautical measure was defined the way it was — a nautical mile is a minute of latitude, which made a chart’s own graticule into a distance scale, and which is the practice a scale bar is right in one place recommends returning to. A navigator reading distance off the side of a chart rather than off a bar in the corner is using the one degree measure that behaves like a length.

The definition has since been fixed at 1,852 metres exactly, which is a convention rather than a measurement and is within 0.1 per cent of a minute of latitude at 45°. So the unit that was built on this exception has quietly become a unit that no longer depends on it — the same move the units are part of the coordinate records for the survey foot, and for the same reason: a unit defined by a measurement of the Earth has to be redefined the moment the measurement improves.

The nautical mile’s own arithmetic is worth finishing, because it is the sharpest case of a unit built on this exception and then cut loose from it. A minute of latitude runs from 1,842.90 metres at the equator to 1,861.57 at the pole, and at 45° it is 1,852.196. The defined mile of 1,852 metres is therefore short of a minute of latitude at 45° by 19.6 centimetres — one part in 9,450, or 0.011 per cent, which is a good deal closer than the tenth of a per cent usually quoted for it.

That closeness is the whole design. The definition was chosen to sit at the middle of the range rather than at either end, so the unit is never wrong by more than half the meridian’s own variation: 0.49 per cent short at the equator and 0.52 per cent long at the pole. A navigator reading distance off a chart’s side is therefore using a scale that is correct to half a per cent everywhere, which is finer than any other property of a chart, and finer than the plot of a fix.

And the redefinition removed a dependence rather than a discrepancy. Fixing the mile at 1,852 metres exactly means the unit no longer moves when the ellipsoid does — a change of reference figure now changes what a minute of latitude is and does not change what a nautical mile is. The two quantities were one thing and are now two, agreeing to a part in nine thousand, and the agreement is a fact about 1929 rather than a fact about the Earth.

Where this ladder goes #

A degree misused as a length is the smallest case of the field’s general problem: an operation performed in the coordinates rather than on the surface. The next rung takes the same failure to a quantity where it is a factor of three rather than a factor of two — the area of a polygon, which needs a surface before it can be computed at all.