Which small quantity the series is in

Nine rungs of this ladder measure what the ellipsoid does to a coordinate, and every one of them computes through a series. The meridian arc is a series. The transverse Mercator is a series. The conformal latitude is a series, and so is its inverse.

Every one of those series is written in the third flattening, n = f/(2 − f). Nothing in this collection says why, because the choice was inherited rather than made: Krüger wrote the transverse Mercator that way in 1912, every modern implementation followed, and the parameter arrives with the formula.

It is worth measuring, because it is a constant in the machinery that nobody has priced.

Three numbers for one shape #

An ellipsoid of revolution has two parameters and any number of ways to write the second one. The three that appear in the literature are

$$f = \frac{a-b}{a}, \qquad e^2 = \frac{a^2-b^2}{a^2} = 2f - f^2, \qquad n = \frac{a-b}{a+b} = \frac{f}{2-f}.$$

They describe the same shape and they are not the same size.

e² is 3.99 times n. If a series truncated after k terms has an error of about the parameter to the (k+1), then the same number of terms buys 3.99⁴ ≈ 253 times the accuracy in the smaller parameter, and the same accuracy costs about half as many terms.

That is the folklore, and it is nearly right.

What was computed, and how #

Two expansions of one integral, and a third thing that is neither.

The integral is the definition: the meridian arc from the equator to latitude φ is

$$M(\varphi) = a(1-e^2)\int_0^\varphi (1 - e^2\sin^2 t)^{-3/2}\,\mathrm{d}t,$$

evaluated here by Simpson’s rule on four thousand panels. It shares no algebra with either series, which is what makes it a reference rather than a restatement.

The n series is the one this site already uses: Krüger’s, with coefficients in n up to the fourth power.

The e² series is Bessel’s, written out from the standard coefficients rather than derived, so that the two expansions are genuinely independent pieces of code:

$$M = a(1-e^2)\left[A_0\varphi - A_2\sin 2\varphi + A_4 \sin 4\varphi - A_6\sin 6\varphi + A_8\sin8\varphi\right].$$

At full order the two agree with each other to 9 × 10⁻⁵ metres over the whole meridian, which is the first check: two independent expansions of one integral, agreeing, before either is compared against anything.

The advantage arrives at the third term #

At one term the two are identical to five figures and the e² series is fractionally the better of the two. At two terms it is still fractionally ahead. The n series does not win until the third term, and it does not win by the predicted factor of four per term — at four terms it is ahead by 1,175, which is far more.

Both departures from the folklore have the same cause and it is worth stating carefully, because it is what makes this a measurement rather than an illustration.

Why the two agree at low order. The leading behaviour of a meridian arc is the same whichever parameter it is written in — it is the same function — so the first terms of the two expansions are numerically almost the same thing rearranged. The expansions only part company once the terms that distinguish them arrive.

Why the n series then wins by more than four per term. It does not. What happens at four terms is that the e² series stops improving: its coefficients are carried only to e⁸, so the last point is a floor set by the coefficients rather than a truncation set by the sine terms. The 1,175 is a comparison between a converging series and one that has run out of algebra, which is the honest reading and is also the reading that matters in practice — an implementation carries the coefficients somebody wrote down.

The rate is the parameter, fitted #

The check that these are what they are claimed to be — expansions in their stated parameters, rather than two approximations that happen to work — is that each one’s error should fall by a factor of its own parameter per term.

Fitting the slope of log(error) against order gives 1.630 × 10⁻³ for the n series against n = 1.679 × 10⁻³, which is 0.5 per cent. The e² series fits to 1.51 × 10⁻², against e² = 6.69 × 10⁻³, and is out by a factor of two — because its last point is the floor described above and a floor flattens a log-linear fit.

A check that fails for a reason is more useful than one that passes. The e² series’ fitted rate is wrong, the reason is known, and the reason is the same fact the table’s last row reports.

The same choice inside the transverse Mercator #

The meridian arc is the simplest place the choice appears and not the place it matters most. Every national grid in the world computes through a transverse Mercator, and the series that does it is a much larger object with the same decision inside it.

Transverse Mercator and the series that computes it sets out the construction; where the series stops being the map prices what happens outside the zone it was written for. Neither asks what the series is a series in, and the answer is the same n, for the same reason and with a larger stake: a national grid is used at the millimetre by people who never see the formula.

The one thing the transverse Mercator adds is that the coefficients are polynomials rather than numbers, so the “coefficients run out” failure the e² meridian series shows has a second form: a coefficient carried to a lower power of n than its neighbours puts a floor under the whole series, and it is invisible because every individual term looks correct.

Where the model stops #

A series is a statement about a regime, and the regime is not stated with it. Everything above holds because both parameters are small on the Earth. On a fast-rotating minor body with a flattening of a third — a real shape, and one a map of a body with three axes has to deal with — n is 0.176 and e² is 0.51, and four terms are out by 674 metres and 229 kilometres respectively.

That is the refusal the rung needs. If the n series had stayed accurate on a body it has no business being accurate on, the whole comparison would be measuring something other than convergence.

The comparison is of one quantity. The meridian arc is the simplest ellipsoidal series and the one both traditions write down identically. A transverse Mercator series is a much larger object with the same choice inside it, and whether the ratio is the same there is not measured here.

And the coefficient tables are somebody’s. Both series are used at the coefficients their tradition publishes rather than at coefficients derived here, which is deliberate — the comparison is between what an implementer actually types — and it is why the e² series has a floor at all.

Why a smaller parameter is not automatically the right one #

The obvious next move is to look for a still smaller parameter, and it is worth saying why that does not work, because the reasoning is what makes n the end of the sequence rather than a step in it.

Any quantity that is a fixed multiple of the flattening — half of it, a quarter of it — makes the series converge faster per term by exactly that factor and makes every coefficient larger by the same factor. Nothing has been gained: the terms of the series are products of coefficient and parameter, and rescaling the parameter rescales the coefficients back.

What makes n genuinely better than e² is not that it is a smaller number. It is that the meridian arc, written in n, has coefficients that are bounded — the trigonometric series in n has coefficients of order one, so a term really is of order n^k — while the same function written in e² has coefficients that grow. Compare the two leading corrections: in n the sine coefficients are −3/2, 15/16, −35/48, 315/512, which hover around one; in e² they are 3/8·e², 15/256·e⁴, 35/3072·e⁶, and each carries a growing polynomial in e² inside it.

The parameter is doing the work only if the coefficients stay put, and checking that they do is the part of “expand in a small quantity” that a fitted convergence rate actually tests. The n series’ fitted rate matching its parameter to half a per cent is that test passing.

What it costs to get this wrong #

The practical consequence is a millimetre question, and millimetre questions are where this site’s subject usually lives.

Reaching a millimetre on the meridian needs three sine terms in either series. Reaching a micrometre needs four in n and is unreachable in e² at published coefficients. So an implementation working to the precision a published coordinate is a result describes — where a national adjustment quotes millimetres and means them — is inside the regime where the choice does not matter, and one working to the precision a coordinate is the output of a solve needs for a comparison of two epochs is not.

That is a narrower claim than “always use n”, and it is the one the numbers support.

The check that had to be built first #

Nothing above could be measured until there were two independent expansions in the tree, and building the second one is most of what this rung cost.

The temptation was to derive the e² coefficients from the n ones by substitution, which is a page of algebra and would have produced two series sharing every step of their derivation. That is not a comparison; it is one series printed twice, and any error in the shared step would have cancelled out of both sides.

So Bessel’s coefficients are typed from the standard form instead, with the risk that goes with typing coefficients, and the risk is bought off by the agreement check: the two full-order series match over the whole meridian to 9 × 10⁻⁵ metres. A typo in any of the fifteen numbers would be a metre or worse, so the agreement is a strong statement about both.

That is the same shape of check as the meridian arc against its own integral — a series against a quadrature that shares no algebra with it — applied one level up, between two series rather than between a series and an integral.

The last row of the order table deserves a sentence of its own, because it is where a reader would stop reading. At four sine terms the n-series is at 7.6 × 10⁻⁸ metres over the whole meridian, which is seventy-six nanometres — smaller than the wavelength of visible light, on a quantity ten thousand kilometres long. Nothing in geodesy needs that, and the reason the series is carried to four terms anyway is that the fourth term costs one multiplication and the alternative is having to know when three would do.

The generalisation #

The pattern is one the collection meets whenever a formula is inherited: a constant in a piece of machinery is a decision that somebody made and nobody has re-examined.

The scale factor was chosen makes the same point about a grid’s 0.9996. Sixty zones was a decision about one latitude makes it about UTM. The tolerance sits sixty times above the noise floor makes it about this site’s own conformality test.

What is different here is that the inherited decision turns out to be right, for a reason that can be stated in one line, and that the reason is not the one usually given. The third flattening wins because it is four times smaller, and it wins by a thousand rather than by two hundred and fifty because the alternative’s published coefficients run out first.

Who found it, and when #

Bessel published the meridian arc in e² in 1837 and it is the form every nineteenth-century geodesy text carries. Krüger’s 1912 transverse Mercator is written in n, and Helmert had already used the third flattening for the meridian arc before him; the modern accounts of the transverse Mercator series — Karney’s in particular — state the advantage of n explicitly and give the convergence in terms of it.

What is unusual about measuring it here is only that the two are run side by side against the same reference, at the same orders, with each one’s rate fitted rather than quoted. The result confirms the folklore in the direction it points and corrects its arithmetic in two places, which is roughly what a measurement of a well-known thing is for.

What a measurement of a known thing is for #

The closing description of this rung’s contribution is unusually modest and worth taking seriously, because most of what a careful collection does is of this kind.

The folklore was right about the direction. The third flattening is the better expansion parameter, everybody in the field says so, and the measurement agrees. Nothing was overturned.

It was wrong about the arithmetic in two places, which is what running the comparison side by side finds and what quoting cannot. A rate that is asserted rather than fitted carries whatever error the assertion had, and the error survives being repeated because each repetition is of the assertion rather than of the thing.

And the value is in the fitting rather than the answer. Both series were run against the same reference, at the same orders, with each rate measured from the output — so the comparison is a measurement of two methods rather than a restatement of a preference between them. That is the difference between knowing that n is better and knowing by how much and at what order the advantage appears.

And the audit needed a piece of machinery built first, which is the other half of what such an exercise costs. Comparing two series against each other says only which is closer to the other; comparing both against an independent reference says which is closer to the truth, and the reference had to exist before either rate meant anything. That is the usual shape: the measurement is cheap and the thing it is measured against is the work.

Which is the ordinary yield of auditing something well known. Most of the time the folklore survives, slightly corrected, with numbers attached where it had adjectives — and the numbers are what let somebody decide whether the advantage is worth the change in their own case, which the adjective never could.

The reference here is the exact computation, evaluated to well beyond the accuracy either series claims, and it is what makes both fitted rates measurements rather than comparisons. Without it the two series would agree with each other where they are both accurate and diverge where they are both wrong, and the divergence would look like a result. That is the specific hazard in auditing two approximations against each other, and the only defence is a third thing that is not an approximation of the same kind.

It is also the reason this rung’s numbers can be quoted at all. It is also why the rung could correct the folklore’s arithmeticIt is also why the rung could correct the folklore’s arithmetic rather than merely restating its preference: a preference can be checked against another preference, and an arithmetic claim needs something exact to be checked against.

Where the ladder goes next #

Ten rungs price what an ellipsoid does to a coordinate, and this one prices the arithmetic that computes it. What is missing is the other end of the same question: an ellipsoid is a fit, its two parameters were estimated from observations, and they have an uncertainty and a correlation. Every series above is exact to a nanometre in parameters that are themselves known to a few parts in a billion, and how those two error budgets compare is a question none of the ten has asked.