The correction is the smaller of the two corrections

An angle is a difference, and the difference doubles the error priced what a substituted sphere does to the three angles of a geodesic triangle, and the numbers were large: eight and a half minutes of arc on the geodetic-latitude sphere, on a thirty-kilometre triangle at 45° north. It also left a question standing, and the question is the one a computer of the eighteen-forties would have asked first.

Nobody solving a triangle on a sphere solves it by taking the angles at face value. They apply Legendre’s theorem: subtract a third of the spherical excess from each of the three angles, and solve what is left as a plane triangle with the same sides. The theorem is from 1787, it is exact to the order anybody needed, and it is the whole reason a spherical triangulation could be computed with logarithm tables — it converts a problem in spherical trigonometry into one in plane trigonometry, which is what what a tape measures has to be reduced to before any of it becomes a coordinate. It is also a correction to the same computation the sphere choice corrupts.

So there are two corrections in play and nothing so far has said which is the larger. That is the question here.

Two terms that do not scale together #

The excess is a statement about area. A geodesic triangle on a sphere of radius $R$ has an angle sum exceeding $\pi$ by its area divided by $R^2$, so a triangle of side $s$ has an excess proportional to $s^2$. On 60° triangles at 45° north the measured values run from 0.0022 arcseconds at one kilometre to 351 arcseconds at four hundred — a factor of a hundred and sixty thousand for a factor of four hundred in the side, which is the square law to within the third digit.

The error a substituted sphere leaves is a different kind of quantity, and a bearing on a sphere is decided by its latitude, not its radius is where that became clear. Four of the six substitutions — geodetic, parametric, authalic, rectifying — put a latitude on the sphere that differs from the ellipsoid’s geodetic latitude at first order in the flattening. The difference is a property of the corner, not of the distance between corners, so it does not shrink when the triangle does. The geodetic-latitude sphere carries 520.69 arcseconds into a one-kilometre triangle and 521.09 into a thirty-kilometre one and 525.98 into a four-hundred-kilometre one: flat to a part in two hundred over a range of four hundred in the side.

Two of the six are different. The conformal and geocentric spheres agree with the ellipsoid at first order, so the first-order floor is not there — which is the same distinction which small quantity the series is in draws between an expansion in the flattening and one in the separation; what is left is a second-order term that behaves like an ordinary truncation error and grows with the separation of the points. The conformal sphere reads 0.052 arcseconds at one kilometre, 0.524 at ten, 5.205 at a hundred, 20.329 at four hundred — proportional to the side across three decades.

A flat line, a line of slope one, and a parabola. Two of the three pairs cross once each, and the third — the conformal sphere against the excess — is a genuine contest between a line and a parabola, which is the case worth having.

Where they cross #

The crossings, for 60° triangles at 45° north turned 45° from the meridian — the configuration the angle measurement published its numbers at, kept here so the two can be read against each other:

The ordering is the reverse of the ordering of the errors, and that is forced rather than surprising. A term growing as the square of the side overtakes a small constant early and a large constant late; the better the sphere, the sooner the excess is the larger of the two things a computer is correcting for. Bessel’s sphere, which moves the longitude as well as the latitude and is exact, has no crossing at all — its worst angle at eight hundred kilometres is nine parts in $10^{11}$ of an arcsecond against an excess of 1,406, and there is nothing for the excess to overtake. That is the case the instrument has to get right to be worth anything, and it is checked before any crossing is quoted.

The interesting number is where a real chain sits. A first-order triangulation ran on sides of twenty to fifty kilometres. That band lies above the geocentric sphere’s crossing, across the conformal sphere’s, and an order of magnitude below the other four. So the answer to “which correction matters more” is not one answer: it depends on which sphere the computer chose, and for the best available choice it depends on the individual triangle.

One of the two announces itself and the other does not #

The comparison so far is between two sizes, and it leaves out the thing that decides which correction a computer actually applies.

The excess is visible from inside the computation. A surveyor observes three angles. Their sum exceeds 180° by the spherical excess — that is not a theorem about the triangle, it is what the observations say, and it is how the excess was found in practice, by adding up what was measured. On the thirty-kilometre triangle the sum comes to 180° 00′ 01.98″, and the 1.98″ is there on the page in the observing book before anybody computes an area. A computer who forgot Legendre’s theorem would be reminded of it by their own angle sum.

The sphere error is invisible from inside the computation, and that was settled before this. The three angle errors a substitution leaves very nearly cancel round the triangle: on the geodetic-latitude sphere the individual angles are wrong by 521, 516 and 5 arcseconds and their sum is wrong by less than a ten-millionth of one. So the angle sum reads exactly the same whichever sphere the computer chose. The closure check — the one instrument that reads the observations back and asks whether they are consistent — cannot see the larger of the two terms at all, and what a closed figure cannot see is the general form of that failure.

Put the two facts beside each other and the historical shape of the thing falls out. The correction that is small, computable and self-announcing was applied universally, explained in every textbook and named after its author. The correction that is one to two hundred and fifty times larger was a choice made once, at the top of the computation, in selecting which sphere the chain would be reduced to — and nothing downstream would ever mention it again. That is not carelessness. It is the ordinary consequence of a check that is blind in exactly one direction, and it is the same shape as a traverse must close: a figure that closes is consistent with itself, which is a weaker statement than a figure that is right.

There is a second consequence, and it is the one that makes the crossing worth locating rather than merely interesting. Because the excess is observed rather than modelled, applying Legendre’s correction costs nothing in assumptions — it uses a number the observations already contain. Choosing a sphere costs an assumption about the figure of the Earth, and an ellipsoid computed to a nanometre is known to a decimetre is about how far that assumption can be trusted. So the two corrections are not merely different sizes; they are different kinds, and the larger one is the one that depends on something the survey cannot measure for itself.

The same question asked of a length #

An angle error is not what a surveyor is paid to control. A triangulation exists to produce coordinates, and an angle reaches a coordinate through the solution of the triangle: measure one base, take the three angles, subtract a third of the excess from each, and solve the plane figure by the sine rule. So the honest form of the comparison is in metres of computed side.

That reformulation is not cosmetic, and the reason is worth stating before the numbers. The plane sine rule does not weight the three corners equally. An error in the angle opposite the side being computed moves that side hardly at all; an error in the angle the measured base subtends moves it by the full cotangent. The worst of the three angle errors is therefore not the quantity a length is sensitive to, and there is no reason for the two comparisons to cross in the same place.

On the thirty-kilometre triangle:

• omitting Legendre’s correction moves the computed side by 166 millimetres;
• the conformal sphere moves it by 228 millimetres;
• the geocentric sphere by 156 millimetres;
• the rectifying sphere by 10.9 metres;
• the geodetic-latitude sphere by 43.0 metres.

The last number is the one to sit with. Forty-three metres in thirty kilometres is 1,435 parts per million — a distortion two orders of magnitude beyond anything a first-order survey would accept, produced not by a bad observation but by a choice of auxiliary sphere, in a computation that also carefully applies a correction worth sixteen centimetres.

And the two good spheres land where they have to for the question to be interesting at all: 228 and 156 millimetres against a correction worth 166. Same order, either side of it. If the second-order spheres had come out at a micron or at a kilometre there would be nothing to measure; that they come out level with Legendre’s own term is what makes the crossing a real location rather than a formality.

The two crossings disagree, and by how much is the finding #

For the four spheres wrong at first order the two crossings are close — within seven per cent for rectifying, authalic and parametric, and thirteen per cent for geodetic. Their angle errors are large and nearly equal at all three corners, so the sine rule’s unequal weighting has little to work with.

For the two spheres right at first order the disagreement is large and it goes both ways:

• the geocentric sphere crosses at 17.6 kilometres asked of the angle and at 8.0 asked of the side — a factor of 2.2 earlier;
• the conformal sphere crosses at 23.8 kilometres asked of the angle and at 41.3 asked of the side — a factor of 1.7 later.

Two spheres whose angle errors sit within thirty per cent of each other at thirty kilometres, and whose length crossings are five times apart. The second-order residues are small enough that their distribution around the triangle decides the answer rather than their size, and the distribution differs between the two substitutions. This is the same lesson what a closed figure cannot see reaches from the other side: a check that reduces three numbers to one throws away the part that was going to matter.

It also qualifies the table above rather than replacing it. Which sphere is better has the same answer either way — Bessel’s, then conformal or geocentric, then the rest. At what size the correction takes over does not, and a paper quoting one crossing without saying which question it answered has quoted a number with a free choice inside it, which is the defect the tolerance that decides the verdict catalogues on the other side of this collection.

Where the crossing moves #

Two things move it, and one of them is much larger than the other.

Latitude splits the two families apart. The four first-order substitutions carry an error falling as the square of the cosine of the latitude, which was measured for the angles themselves before it was measured for the crossing, so their crossings fall with them: the geodetic sphere crosses at 685 kilometres on the equator and 181 at 75° north. The two second-order substitutions carry the $\sin 2\varphi$ signature instead. It vanishes at the equator and at the pole, where every auxiliary latitude coincides with the geodetic one, and peaks between them; so their crossings peak too, at 23.8 kilometres for the conformal sphere at 45°, falling to 12 at 75°, and on the equator there is no crossing at all because the sphere is exactly right and nothing is left for the excess to overtake.

The two families therefore move apart towards the equator and together towards the pole. At 75° north the conformal sphere’s crossing is fifteen times the geodetic’s; on the equator the ratio is unbounded. A national survey in Scandinavia is choosing between substitutions whose crossings are closer together than one in the tropics is, which is a statement about how much the choice costs rather than about which choice is right.

Shape moves it less, but not negligibly. The excess is proportional to area, so a sliver at 20° has to grow nearly four times as large as a right-angled triangle before the excess catches the same sphere: 56.8 kilometres against 15.2 for the conformal substitution. A chain of slivers is further inside the sphere’s regime than a chain of well-conditioned triangles — which is the opposite of the usual advice about triangle shape, and for a different reason. Slivers are avoided because they condition the solve badly, not because of anything to do with the sphere; this says that the one thing a sliver is good at is hiding the excess. The scale factor of a line makes the neighbouring observation about a grid: a quantity that depends on where a line runs rather than on how long it is behaves differently under every change of shape.

What each number was checked against #

The instrument is small and the two quantities it compares are computed by different routes, so the controls are about the scaling rather than about agreement with a reference.

The excess must grow as the square of the side. Measured from 3 km to 200 km, it grows by a factor of 4,444.0 against a square law’s 4,444.4 — within a part in ten thousand. If it did not, the crossing would be an artefact of where the bracket was placed rather than a property of the two terms.

The first-order spheres must be flat in the side. Each of the four is required to drift by less than two per cent between a one-kilometre triangle and a fifty-kilometre one. The geodetic sphere drifts by 0.13 per cent. A substitution whose error grew with the triangle would not be a first-order latitude error, and the classification of the six spheres into two families would be wrong.

The conformal sphere must grow in proportion to the side, checked as a factor between eight and twelve for a tenfold increase. It gives 9.94. Without that the contest between a line and a parabola is not what is being drawn.

A sphere that never loses must report no crossing. Bessel’s sphere is that case, and the bisection is required to return nothing for it rather than a large number. A search that always finds a crossing would find one for an exact substitution too, and the whole table would be an artefact of the bracket’s upper end.

The ordering of the crossings must reverse the ordering of the errors. The conformal sphere’s crossing is required to be less than a tenth of the geodetic sphere’s. It is one twentieth.

The two crossings must differ. At least one sphere is required to move by more than twenty per cent between the angle question and the length question, because if the sine rule weighted the three corners equally the whole of the second half of this essay would be measuring nothing. The geocentric sphere moves by a factor of 2.2.

Where the model stops #

One triangle, not a network. Every number here is the first triangle of a chain solved from a measured base. A real triangulation carries the error forward through fifty or a hundred triangles and adjusts the whole figure to close, and the weights are a guess the solve believes and the network’s answer is decided before it is measured are both about what an adjustment does to a systematic error it cannot see. A 1,435-part-per-million scale error common to every triangle does not misclose; it produces a network that is the right shape and the wrong size, which is exactly the failure a baseline at the far end was invented to catch. The two ways to spread a misclosure is about what an adjustment does with a discrepancy it can see, and the point here is that this one never becomes a discrepancy.

The base is taken as exact. A measured baseline of the period was good to a part in a million at best, so on the thirty-kilometre triangle it contributes about thirty millimetres — below the correction and far below every sphere but the two good ones. That ordering would reverse for a modern electronic distance measurement, where the base is good to a part in $10^7$ and the conformal sphere’s 228 millimetres is the whole budget.

Legendre’s theorem is taken as exact. It is not: it has its own truncation, of order $s^4$ relative to the leading term, which on a hundred-kilometre triangle is a few parts in $10^7$ of the excess and is below everything drawn here. On a thousand-kilometre triangle it would not be, and the comparison would need a third curve.

And the substitutions are taken as the six an auxiliary-latitude table offers. A computer free to choose a sphere by fitting it to the region — the thing a datum is fitted to a region does for the ellipsoid itself — could beat all six over a small enough area, and nothing here prices that option. The six are the ones a nineteenth-century handbook contained, and four radii of the Earth priced them for a distance, which is what makes them the right set for a question about what those computations actually cost.

Still open: whether the crossing is visible in a surviving network #

The angle measurement ended by asking whether any of this reached a published coordinate, and the crossing sharpens that question rather than answering it. A scale error of 1,435 parts per million would be unmistakable — it is fourteen metres in ten kilometres, and a re-observation would find it immediately and attribute it to the base. The two good spheres are the realistic case, and there the error is 228 millimetres in a thirty-kilometre triangle, or about eight parts per million.

Eight parts per million is the interesting size. It is below what a nineteenth-century baseline could have detected and above what a modern re-observation of the same network would miss, so a systematic scale difference of that order between an old triangulation and its modern replacement is exactly what a wrong choice of auxiliary sphere would leave behind. Several national networks have documented scale differences in that band and attribute them to baseline standardisation.

What would settle it is not another computation on a stated triangle. It is the pattern: a sphere error runs as $\sin 2\varphi$ for the good substitutions and as $\cos^2\varphi$ for the bad ones, and a baseline error does neither. A network observed over ten degrees of latitude carries enough of that pattern to separate the two, and whether any surviving adjustment does is a question about an archive rather than about geometry.