An ellipsoid computed to a nanometre is known to a decimetre

Which small quantity the series is in measured the meridian series on WGS84 and found it exact to 7.6 × 10⁻⁸ metres at four sine terms — seventy-six nanometres, on a distance of ten thousand kilometres, smaller than a wavelength of light. Four radii of the Earth went on to price the sphere that stands in for the ellipsoid, in parts per million. Both measure how well a computation reproduces an ellipsoid. Neither asks how well the ellipsoid reproduces the Earth, and a nanometre of agreement with something is only as useful as the something.

There are two different objects in geodesy called an ellipsoid, and the difference between them decides what that nanometre is worth.

Two things called an ellipsoid #

The first kind is defined. WGS84 fixes its semi-major axis at exactly 6,378,137 metres and the reciprocal of its flattening at exactly 298.257223563, and the ellipsoid is a level surface shows that everything else it publishes follows from those two and two more. None of the four is a measurement in the sense that matters. They were chosen, with the Earth in view, and once chosen they are exact: a distance computed on WGS84 has no uncertainty from WGS84’s parameters, because the parameters are what WGS84 is.

GRS80, the reference system most national datums are built on, is defined the same way with one difference in what it chose to fix. It fixes the same axis, and instead of a flattening it fixes the Earth’s dynamical form factor — the coefficient that says how much the planet’s gravity departs from a sphere’s — so its flattening is derived, and comes out as 298.257222101. WGS84 adopted a flattening directly from a coefficient rounded when it was defined, and the two differ in the ninth significant figure. Neither is wrong. They are two definitions.

The second kind is fitted. The ellipsoid that best fits the Earth’s actual shape is estimated from satellite orbits, altimetry over the oceans and gravity, and the International Earth Rotation and Reference Systems Service publishes it in the table of numerical standards in its Conventions of 2010: a semi-major axis of 6,378,136.6 metres, uncertain by 0.1 metre, and a reciprocal flattening of 298.25642, uncertain by 0.00001. That is a measurement. It has error bars, and it is not the same pair of numbers as either convention: its axis is forty centimetres shorter than WGS84’s.

So a meridian distance computed on WGS84 carries three separate budgets that are routinely run together. The series has a truncation error — the difference between the computation and the definition. The best-fitting ellipsoid has an uncertainty — the difference between the definition of the Earth’s shape and the Earth. And the conventions disagree with the best fit and with each other — differences that are not errors of anybody’s at all.

The series is inside the ellipsoid’s uncertainty from its second term #

The three are different kinds of quantity, but all three are lengths, and on one meridian distance they can be put on one scale.

With no sine terms at all, the series is out by sixteen kilometres at its worst. One term brings that to 16.8 metres; two to 22 millimetres; three to 31 micrometres; four to 76 nanometres. The best fit’s uncertainty over the same distance is 15.7 centimetres, and the series falls inside it at the second term.

Everything the third and fourth terms add, then, is accuracy about WGS84 rather than about the Earth. Carried to four terms, the series reproduces the distance WGS84 defines to a precision two million times finer than anybody knows the distance on the Earth that WGS84 was chosen to approximate. The last two terms are arithmetic about a convention.

That is not a reason to drop them, and the reason it is not is the most useful thing the three budgets say.

Exact about a convention is what makes computations agree #

A coordinate is not published as a claim about where a point is on the Earth to a nanometre. It is published as the result of a stated computation on a stated definition — a published coordinate is a result is the essay about what that means for a survey mark — and the only thing that can check such a result is another computation of the same thing. Two implementations of the transverse Mercator on the same ellipsoid should agree to far below any survey’s precision, because if they do not, a disagreement between two grid coordinates cannot be attributed to the ground.

So the series is carried past the Earth’s own uncertainty for the same reason a round trip is tested to a fraction of a millimetre. The inverse of the series is not the series of the inverse found that a forward and a backward series at the usual order do not compose to the identity, and put the failure at 0.39 millimetres. That is a defect of the arithmetic, and it was only visible because the arithmetic was held to a standard set by other arithmetic rather than by the Earth. A floor of seventy-six nanometres is what lets every larger discrepancy be read as something other than truncation.

The budgets separate two statements that the word accuracy runs together. The series’ precision is a statement of consistency: every computation on WGS84 gets the same answer. The best fit’s uncertainty is a statement of knowledge: nobody’s computation gets the Earth closer than that. Neither implies the other, and a map that quotes one as though it were the other has confused them.

Nearly all of the fit’s uncertainty is in its axis #

The best fit’s two uncertainties are quoted in very different forms — a tenth of a metre on the axis, a hundred-thousandth on the reciprocal flattening — and it is not obvious which matters more for a distance.

The axis dominates by a factor of 279, and the reason is structural rather than a matter of which was measured better. At a fixed flattening, every distance on an ellipsoid is exactly proportional to its axis, so a relative uncertainty of sixteen parts in a billion in the axis is sixteen parts in a billion in every distance. The flattening enters a meridian distance only as a correction to that: a relative change in the flattening moves the distance by half the flattening times as much as the same relative change in the axis — about a six-hundredth. Its thirty-four parts in a billion become about fifty-six parts in a trillion of the distance.

The table states no correlation between the two uncertainties, so they are combined here as independent. It hardly matters: with one of them 279 times the other, even a perfect correlation between them would change the total by less than a third of a per cent. The Earth’s ellipsoid is uncertain in its size, not in its shape.

That puts the figure of the Earth was measured in a new light. The eighteenth-century expeditions to Lapland and Peru were after the flattening, because the flattening was the open question, and the size was taken as roughly settled. Two and a half centuries later the flattening is still the less precisely known of the two in relative terms, and its uncertainty matters 279 times less to a distance than the size’s does.

Two conventions a tenth of a millimetre apart, and the Earth four decimetres from both #

The disagreement between the conventions is the third budget, and its two parts could hardly be more different in size.

WGS84 and GRS80 differ by a hundred and five micrometres in their polar axes and by nothing at all in their equatorial ones. That is the figure every geodesy text quotes when it says the two are the same ellipsoid for practical purposes, and the three budgets say precisely what “practical” means: the difference is some fifteen hundred times smaller than the uncertainty of the ellipsoid either was chosen to approximate. No measurement of the Earth could say which of the two is closer to it.

The best fit sits forty centimetres inside WGS84 at the equator and forty-six at the poles. That is not an error in WGS84. WGS84 was fixed in 1984 to the best knowledge of the time and has kept its axis and flattening since, because a reference system whose axis changed with each new estimate would move every coordinate expressed on it. The distance between a convention and the current best estimate is the price of that stability, and it is paid deliberately.

The same coordinates name two different points #

The practical meaning of forty centimetres is easiest to see by putting a coordinate on both ellipsoids.

A latitude and longitude with zero height is a point on the surface of an ellipsoid, and on these two ellipsoids the same numbers name points forty to forty-six centimetres apart. Almost all of that is vertical: 40.0 centimetres at the equator, 42.8 at 45° and 45.6 at the pole, against a horizontal part that is zero at the equator and the pole and largest at 45°, where it is 5.6 centimetres. So a height of zero above WGS84 is a height of forty-odd centimetres above the Earth’s best ellipsoid, while the two readings of a latitude and longitude put a point in nearly the same place on the ground.

That vertical gap does not show up anywhere as an error, because every quantity that depends on it is published relative to a stated ellipsoid. A geoid model gives the height of the geoid above a named reference ellipsoid, usually GRS80 or WGS84, so the forty centimetres between that ellipsoid and the best fit is absorbed into every geoid height the model contains. A levelled height and a satellite height agree through the geoid model, and neither needs to know that the ellipsoid underneath both of them is not the one that fits the Earth best. Height above what is the essay about how many different things a height can be above, and this is the smallest and least discussed of them: two ellipsoids, both called the Earth’s.

This is the same shape of problem as datum shifts dwarf projection errors, which found hundreds of metres between the datums a coordinate might be on and parts per million between the projections it might be drawn in. The difference is where the numbers come from. A datum shift is a difference between two realisations of position on the ground. The forty centimetres here is a difference between two definitions of a surface, neither of them tied to any point — and it is a thousand times smaller than a historical datum shift and millions of times larger than the series error computed on either surface.

Along the meridian the order holds #

The comparison has been made on the whole distance from equator to pole. Shorter distances scale the budgets differently, and whether their order survives is worth checking rather than assuming.

The fit’s uncertainty and WGS84’s distance from the best fit both grow with the distance, because both are dominated by the axis, and a difference in the axis is a fixed fraction of every distance. The disagreement between WGS84 and GRS80 behaves differently, because it comes from the flattening alone: it rises to 124 micrometres at 55° and falls back to 82 at the pole, because most of a meridian distance’s dependence on the flattening is carried by its first sine term, which is largest at mid-latitudes and returns to zero at the pole. The series at four terms does not grow at all. It oscillates with latitude, as a truncated sine series does, and never comes within 300,000 times of the fit’s uncertainty.

So the order of the budgets is not a property of the pole. At every latitude a meridian distance on WGS84 is computed far more precisely than the definition is agreed between conventions, which is itself far more precise than the Earth’s ellipsoid is known, which is closer than WGS84 is to it.

The same budgets in parts per billion #

A map’s distortion is usually priced in parts per million, and it is worth putting the three budgets on that scale beside the numbers those essays found, because it shows where each belongs in a sequence of approximations that runs from a sphere to the defining integral.

Substituting a sphere of 6,371 kilometres for the ellipsoid gets a meridian distance wrong by up to 559 parts per million, as four radii of the Earth measured. A transverse Mercator grid shrinks its central meridian by a scale factor of 0.9996 on purpose, which is 400 parts per million. Those two are the size of every question about projections and radii, and they are where the reader of a map should look first.

Below them, by nearly four orders of magnitude, WGS84’s distance from the best-fitting ellipsoid over a quadrant is 67 parts per billion. The best fit’s own uncertainty over the same distance is 16 parts per billion. The disagreement between WGS84 and GRS80 at the pole is eight parts per trillion. And the series at four terms is out by eight parts per quadrillion.

The steps down the list range from a factor of one and a half to a factor of several thousand, and each is a different kind of object: a modelling choice, a design choice, a convention, a measurement, a second convention, an arithmetic. What the sequence shows is that the questions a map reader usually worries about — which radius, which projection, which scale factor — sit four orders of magnitude above the question of which ellipsoid, and that question sits a further seven above the question of how many terms. A computation that gets the first two right and quotes its answer to a nanometre has been careful about the right things in the wrong order of emphasis, but it has not been wrong.

What each budget is for #

The three budgets answer three questions that a coordinate implicitly carries, and it helps to state them as questions rather than as errors.

Is the computation right? That is the series’ budget, and the answer is to seventy-six nanometres. It is the question a software library, a round-trip test and an agreement between two implementations are about, and it is why transverse Mercator and the series treats truncation as an engineering parameter: the order is chosen to put the arithmetic below every other budget by a wide margin, so that it never has to be considered again.

Is the definition the Earth? That is the fit’s budget, and the answer is to sixteen centimetres over a quadrant. It is the question a geophysicist asks, and no amount of arithmetic on WGS84 bears on it. The flattening is not a free parameter showed that the flattening and the Earth’s gravity are tied by Clairaut’s theorem; the best fit is where that tie and the orbits and the oceans agree, and its error bars are how well they do.

Which definition is in use? That is the conventions’ budget, and it is either negligible or large depending on which two are being compared: a tenth of a millimetre between WGS84 and GRS80, sixty-seven centimetres between WGS84 and the best fit. It is not an error at all. It is the information a coordinate needs attached to it, and the forty centimetres at stake is enough to matter to anybody working with heights.

A number quoted to a nanometre can be an answer to the first question and a nonsense answer to the second, and the three budgets make it possible to say which without either overselling the arithmetic or dismissing it.

Still open: a direction rather than a distance #

Every budget here is a length, measured along one meridian. The next quantity an ellipsoid has to get right is not a length at all: the direction from one point to another — the azimuth a geodesic sets out on, which is what a bearing on a map is a stand-in for.

On a sphere that direction has a closed form; on the ellipsoid it has a series, and the sphere of four radii of the Earth that serves a distance well is not obviously the one that serves an angle. How far the azimuth a substituted sphere gives departs from the ellipsoid’s, how that grows with the distance between the points and with their latitude, and whether an angle has its own version of these three budgets, is a question a distance along one meridian cannot ask.