The Earth is a sphere, and when it is not

Every figure on this site before this one was drawn on a sphere, and every caption said so. That is a modelling assumption with a size, and the size is worth knowing before it is relied on again.

The Earth is an oblate ellipsoid. Its equatorial radius is 6,378,137 metres and its polar radius is 6,356,752 metres, so it is 21,385 metres wider than it is tall — a flattening of one part in 298.26.

Three questions, three different answers #

Whether the sphere assumption is safe depends entirely on what is being computed, and the honest answer splits three ways.

For distances, the sphere is fine and the error is a few tenths of a per cent. For the geodesic structure — that the shortest route is not the constant-bearing route, that it runs poleward, that the excess grows with latitude — the sphere is not merely adequate but exact, because those are consequences of positive curvature rather than of any particular shape.

For latitudes, the sphere is wrong by twenty-one kilometres. Not the position of a latitude line, which is a matter of definition, but the relationship between the angle a coordinate reports and the direction of the centre of the Earth.

For conformality, the sphere is wrong outright. Not by a small amount that could be neglected — by an amount that turns the property on and off.

The first: distances #

The corrections run from 0.20 to 0.36 per cent, and the largest is 35 km on a journey of nearly ten thousand. That is negligible for the arguments about routes and decidedly not negligible for anything that has to arrive.

The sign is the part worth noticing, because it means there is no single correction factor. Cape Town to London runs almost due north and comes out shorter on the ellipsoid by 35 km, because the meridians of an oblate body are the short way round it. Anchorage to London runs east–west at high latitude and comes out longer by 24 km. A rule of thumb saying “add a quarter of a per cent” would be wrong in sign for a third of the journeys on this site.

That structure — the correction depending on the direction of travel — is what makes an ellipsoidal geodesic a genuinely harder object than a great circle, and why it has no closed form.

The second: latitudes #

This is where the flattening stops being a correction and becomes the subject.

Latitude on a sphere has one meaning: the angle at the centre. On an ellipsoid it has at least six, and the two that matter most are twelve kilometres apart on the ground at the same point.

Geodetic latitude is the angle between the equatorial plane and the normal to the surface — the direction a plumb line hangs, near enough, and what a levelled instrument measures. Every coordinate anybody handles is this one.

Geocentric latitude is the angle at the centre of the ellipsoid. It is what a spherical treatment implicitly assumes, because on a sphere the normal and the radius are the same line.

At 45° the two differ by 11.55 arcminutes. On the ground that is 21.4 kilometres, which is not a rounding error in any application whatsoever, and confusing them is a specific and consequential mistake rather than a subtlety.

The remaining four — parametric, authalic, rectifying, conformal — are constructions rather than measurements, each defined so that one particular property of the ellipsoid behaves as it would on a sphere. They exist because they make one computation easy, and each of them is a different angle from the one in the coordinate.

Each has a job, and naming the jobs makes the ladder legible rather than arbitrary:

• Parametric (or reduced) latitude projects the point vertically onto a sphere of radius $a$. It is the natural variable for the ellipse itself, and it is what Vincenty’s geodesic method works in.
• Authalic latitude is defined so that equal increments enclose equal areas. Substituting it into a spherical equal-area formula produces the ellipsoidal one, which is how every ellipsoidal equal-area projection is actually computed.
• Rectifying latitude is scaled so that distance along the meridian is proportional to it. It is the variable that turns the meridian arc back into something linear.
• Conformal latitude does the same trick for angles, and is the substitution that makes an ellipsoidal Mercator out of a spherical one.

The pattern is the same in every case: an ellipsoid is a sphere with one property spoiled, and each auxiliary latitude is the change of variable that un-spoils exactly one of them. None of them un-spoils two, which is a small echo of the trade-off the whole subject rests on.

They are ordered, and the ordering is forced. At every latitude the geodetic value is largest, then parametric, authalic, rectifying, conformal, and geocentric smallest. The last two are the interesting pair: they agree to first order in the eccentricity — both are $\varphi - \tfrac12 e^2\sin 2\varphi$ — and separate only at second order, by half an arcsecond at 60°, which is fifteen metres. A treatment that assumed they were identical would be right to within fifteen metres, which is exactly the size of error that survives a review.

The third: conformality #

The sphere assumption does not degrade gracefully here. It fails.

A projection is conformal when it stretches equally in every direction at every point, and “every direction” is measured against the body’s own metric. On an ellipsoid a step of one radian in latitude covers a distance $M\,\mathrm{d}\varphi$ and a step in longitude covers $N\cos\varphi\,\mathrm{d}\lambda$, where $M$ and $N$ are two different radii of curvature that are equal only at the poles.

So a projection built to be conformal on a sphere is not conformal on an ellipsoid, and vice versa. That is exactly what Web Mercator does: spherical formulae, ellipsoidal latitudes, 0.3848° of angular deformation where the correct projection has 1.5×10⁻⁶.

The site now checks that result in both directions, which foundation could not.

Put the whole library on both axes and the two Mercators sit at the far left as conformal, each measured against its own body; Web Mercator, measured against the ellipsoid whose latitudes it consumes, is the single point that is not.

Take the correct ellipsoidal Mercator and measure it against a spherical metric. It fails, at 0.3848° — the same number, with the discrepancy running the other way. Both mismatches are asserted on every build, and the ratio between them is required to stay between a half and two.

That symmetry matters more than it looks. Measured only one way, the machinery is consistent with a rule that says “projections with Web in the name are wrong”. Measured both ways, the claim is the one actually being made: a projection and a metric have to match, and neither is privileged.

The ellipsoid is a definition, not a measurement #

The most common misunderstanding about the ellipsoid is that there is one, and that it is what the Earth is.

There are dozens in current use, they disagree, and the disagreement is deliberate.

The Earth’s actual surface is not an ellipsoid at all. It is a lumpy solid whose gravitational equipotential — the geoid — departs from the best-fitting ellipsoid by up to about a hundred metres, in a pattern with no formula. An ellipsoid is a smooth reference surface chosen to fit a region well, and a national survey run before satellites could only fit its own region.

Airy’s equatorial radius is 643 metres smaller than Clarke’s and 825 metres smaller than Hayford’s, not because any of them is a bad measurement but because each was fitted to a different region and no single ellipsoid fits them all.

Which means the ellipsoid a coordinate refers to is part of the coordinate, and a number quoted without it is incomplete in a way that costs tens to hundreds of metres.

When it is safe to use a sphere #

The practical rule, stated so it can be applied rather than remembered.

Use a sphere for anything about the shape of the problem: whether a route runs poleward, whether a projection can be both conformal and equal-area, how distortion grows with the extent of a region, what a graticule looks like. All of those are consequences of the curvature being positive and none depends on its being constant.

Use an ellipsoid whenever a number will be compared with a measurement on the ground: a survey coordinate, a grid reference, a scale factor, a legal boundary. Also whenever a latitude is being converted into anything, because that is where the eleven arcminutes live.

Never mix them. The failure mode is not an error of one part in three hundred. Web Mercator’s defect is not 0.3% of anything; it is a property that has stopped holding.

Measured along a meridian its angular deformation is not on the noise floor. It peaks at the equator and falls to zero at the poles, which is the signature of a metric mismatch rather than of a projection design — a shape no amount of redesign produces and no amount of care conceals.

Why the flattening exists at all #

Worth a paragraph, because it explains why the number is what it is.

A body that is fluid on a long enough timescale and rotating takes the shape at which gravity and the centrifugal effect balance. Newton worked out in the Principia that this shape is an oblate spheroid and estimated the flattening at about 1/230 on the assumption of uniform density. The Earth is denser at the centre than at the surface, which reduces the effect, and the true figure is 1/298.

So the flattening is a consequence of rotation and of the interior mass distribution, and measuring it was for two centuries the main way of learning anything about the second. That is why the eighteenth-century expeditions mattered: the shape was an argument about physics rather than about maps.

The one place the difference is visible #

Almost nothing about the flattening can be seen. There is one exception and it is a good one.

The equatorial bulge means the Earth’s rotation axis precesses under the Moon’s and Sun’s pull, with a period of about 26,000 years. That is the precession of the equinoxes, it was known to Hipparchus in the second century BC, and it is the flattening’s only large-scale observable consequence — measured two thousand years before anyone knew what caused it.

That figure is the practical face of the whole subject. On a sphere the meridian arc is $R\varphi$ and there is nothing to compute. On an ellipsoid it is an integral, and every survey system in the world is built on a truncated series approximating it.

What the model still cannot carry #

Two things, and both matter for the same reason the flattening does.

The geoid. The ellipsoid is a smooth surface and the Earth’s equipotential is not; the separation reaches about a hundred metres, and it is what makes “height above sea level” and “height above the ellipsoid” different quantities by that much. Every essay here works on the ellipsoid and none of them is about heights.

Time. The plates move at up to about ten centimetres a year, so a coordinate on a plate-fixed datum drifts relative to a globally fixed one at that rate. Over the fifty-year life of a survey mark that is metres. A datum without an epoch is a coordinate without a date.

Both are cases of the same lesson: the reference surface is a model with a stated accuracy, and past that accuracy it is the model rather than the Earth being described.

The sphere is not merely a less accurate ellipsoid. It differs in a way that no choice of radius can repair, because its curvature is constant and the ellipsoid’s is not.

What was computed here #

The flattening, the axes and every derived quantity come from the defining constants of each ellipsoid — a semi-major axis and an inverse flattening — with the eccentricity, the polar radius and the third flattening derived rather than tabulated.

The meridian arc is computed twice, by a series in the third flattening and by Simpson’s rule on the defining integral, and the two are required to agree. They do, to 7.6×10⁻⁸ metres. The first version of the series applied its leading coefficient to every term rather than to the first, which is the obvious way to write it and is wrong by 11 millimetres at 45° — small enough to look like arithmetic noise, and caught only because a second route existed.

The two-sided conformality result is asserted on every build: the ellipsoidal Mercator must pass against the ellipsoid and fail against the sphere, Web Mercator must do the reverse, and the two failures must be within a factor of two of each other in size. They agree to nine significant figures, because they are the same discrepancy measured from opposite ends.

The ellipsoidal geodesics use Vincenty’s iteration, checked against a geodesic obtained by integrating the geodetic reckoning equations with a fourth-order Runge–Kutta step. The two agree to 29 micrometres over journeys of ten thousand kilometres.

What the pictures cannot show #

The flattening. Every cross-section here exaggerates it, and the exaggeration factor is in the caption because the alternative is a figure of a circle labelled as an ellipse — which is what an honest drawing at true scale would be.

The figures also cannot show the geoid, which is the surface the Earth actually has. An ellipsoid is already an idealisation, and the essays that follow are about the idealisation rather than about the planet.

The measurement that settled it #

One part in a hundred, spread over a quarter of the planet, is what two eighteenth-century expeditions were sent to measure — and the inversion from two such arcs back to a flattening turns out to amplify their errors by a factor of 118, which is why the answer was disputed for fifty years. The figure of the Earth was measured follows that arithmetic to the point where it returns a lemon-shaped planet.

Where the sphere costs half a per cent #

The essay’s question is when the spherical approximation stops being good enough. One answer the applied field supplies is unusually clean, because both quantities are closed forms and neither needs a dataset.

The area of a latitude–longitude cell on a sphere is R²Δλ(sin φ₂ − sin φ₁); on the ellipsoid it is the integral of M(φ)N(φ)cos φ, which reproduces the published WGS84 surface area of 510,065,622 km² to the last digit. Over individual cells the two disagree by +0.44 per cent at the equator and −0.79 at 70–80° — while their totals agree to five parts in ten million. A sphere of the mean radius has the right amount of area and distributes it wrongly.

The cost shows in two dimensions and in one. On an equatorial cell the spherical closed form is 0.44 per cent too large, and the three spherical routes agree with each other exactly and with the ellipsoid not at all — which is an approximation showing rather than hiding. In one dimension a sphere gives exactly one for the shape of a small query tolerance at the equator and the ellipsoid gives 0.9933, which is 6,694 parts per million of flattening in a quantity nobody thinks of as geodesy.

Who found it, and when #

Newton argued for an oblate Earth in 1687 from the rotation. Jean-Dominique Cassini and his son measured the meridian across France and concluded the opposite — that the Earth was prolate, elongated at the poles — which set off one of the more expensive arguments in the history of science.

The French Academy settled it by sending two expeditions: Maupertuis to Lapland in 1736 and La Condamine to Peru in 1735, each to measure the length of a degree of latitude at an extreme. A degree is longer near the pole on an oblate body and shorter on a prolate one, and Lapland’s came out longer. Maupertuis returned in 1737 and Voltaire called him the flattener of the Earth and of the Cassinis.

The measurement that settled the shape of the planet was therefore the length of one degree, taken in two places, and it is the same quantity — the meridian arc — that every grid system since has been built on.

It is worth noticing what that makes the flattening: not a shape somebody chose to describe the planet with, but the answer to a measurement that two teams travelled to the ends of the Earth to take, and which came out one way rather than the other.

Where this goes next #

The pair of angles the flattening separates is geodetic against geocentric latitude. The projection every survey system uses on it is transverse Mercator and the series that computes it. And the case that made the ellipsoid this site’s subject rather than a footnote is Web Mercator is not conformal.