What a tape measures

An instrument on a hillside reads 76,895.3 metres to a target. The coordinates of the two points, differenced, give 76,742.0.

The gap is 153 metres, and none of it is measurement error. Four things happen between the reading and the coordinate, three of them are reductions with formulae, and the fourth is a mistake that is easier to make than any of the three.

The three steps #

Slope to horizontal. An instrument measures along the line of sight, which on sloping ground is longer than the horizontal distance by a factor of $1/\cos\alpha$. At 3.2° that is 156 parts per million, and on this line it is 120 metres.

Horizontal to the ellipsoid. Two points 220 metres above the reference surface are further apart than their projections onto it, in the ratio of the radii:

$$k_e = \frac{R}{R + h}$$

which is 42 parts per million here, or 3.2 metres.

The ellipsoid to the grid. Multiply by the line scale factor, which on the British grid at this latitude is about 393 parts per million below unity — 30 metres.

Applied in that order, 76,895.3 becomes 76,742.0 and the disagreement is gone.

The arithmetic, once, end to end #

It is worth doing the whole thing on one line, because the individual terms are easy and the order is not.

A 77-kilometre line, measured at 3.2° of slope on ground 220 metres above sea level in a country where the geoid sits 48.5 metres above the ellipsoid, on the British National Grid:

1. the instrument reads 76,895.3 m along the line of sight;
2. times $\cos 3.2°$, giving 76,775.4 m horizontal — the line has shrunk by 119.9 m;
3. times $R/(R+h)$ with $h = 220 + 48.5 = 268.5$ m, giving 76,772.1 m on the ellipsoid — a further 3.2 m;
4. times the line scale factor 0.99961, giving 76,742.0 m on the grid — a further 30.2 m.

Total reduction 153.3 metres, or about two parts per thousand of the measured length.

Now repeat step 3 with $h = 220$, the levelled height, and the answer moves by 583 millimetres. Nothing else in the calculation changes; no step is skipped; the arithmetic is correct throughout. One input was the wrong kind of height.

The order everybody expects, and the order they come in #

Ask a practitioner which of the three is the largest and the usual answer is the grid reduction, because that is the one with a projection in it and projections are the difficult part.

It is the smallest of the three. The largest is the slope reduction, which is trigonometry, has no cartography in it whatever, and is applied automatically by every instrument built since about 1980.

That inversion matters because attention follows difficulty rather than magnitude. The grid reduction gets the care because it is conceptually hard; the slope reduction gets none because it is conceptually trivial and is handled by firmware. Neither allocation is about how much either term contributes.

The step that is not a step #

The fourth bar in the figure is 583 millimetres, and it is not a reduction. It is the cost of using the wrong height in the second one.

The $h$ in the elevation factor is height above the ellipsoid. What a surveyor has is almost always an orthometric height — carried from a levelling network, read off a map, or taken from a drawing — and the two differ by the geoid separation, which in Britain is around 48 metres and worldwide runs to a hundred.

Using the orthometric height introduces an error of $N/R$, which for $N = 48.5$ metres is 7.6 parts per million. On this line that is 583 millimetres.

Why the ratio is the misleading number #

583 millimetres is 1.9 per cent of the grid reduction it sits beside. As a fraction of the chain it is negligible, and that is exactly how it gets dropped.

But a 77-kilometre line would be expected to close to a couple of centimetres. Against that number — the tolerance the job is actually held to — 583 millimetres is thirty times over.

The assertion in this figure is written to say both things at once, and it took two attempts to get right. The first version demanded that the height mistake exceed five per cent of the grid step, and it failed at 1.9 per cent — because the ratio was the wrong quantity to assert. The repaired version demands that the mistake exceed ten times a survey’s own closing tolerance and stay under a tenth of the grid reduction, which forces the figure to contain the contrast rather than merely a large number.

A correction can be a rounding error relative to the largest term in its own chain and still be thirty times the tolerance of the job. Reading a correction’s importance off its share of the chain is how it gets dropped, and that sentence is what the pair of assertions exists to keep true.

Each step’s own sensitivity #

A useful way to hold the chain is by what each step is sensitive to, because that decides which of them a job has to get right.

The slope reduction goes as $\alpha^2/2$ for small angles, so it is quadratic: doubling the slope quadruples the correction. At 1° it is 152 parts per million, at 3.2° it is 1,560, at 10° it is 15,200. Steep ground is where this term stops being a correction and becomes most of the measurement.

The elevation factor is linear in height at 157 parts per million per kilometre. Nothing about it varies with the line’s length, direction or position, which makes it the easiest of the three to reason about and the one most often carried from a wrong number.

The line scale factor is quadratic in the easting and independent of everything else, which is why the scale factor of a line finds that a line’s direction rather than its length decides how carefully it must be averaged.

Three terms, three different variables, three different powers. No single rule of thumb covers them, and every specification that tries produces the kind of length-keyed advice that is right on average and wrong on any particular job.

The variable the third step depends on runs across the zone rather than along the job. A job on the central meridian and a job at the zone edge have grid reductions differing by fourteen hundred parts per million — more than the whole of the slope reduction on gentle ground — and the two jobs are otherwise identical.

Which height, in more detail #

The distinction the height mistake turns on is the whole subject of height above what?, and its horizontal consequence is here.

An ellipsoidal height is what a satellite receiver produces directly: the distance along the ellipsoid normal from the reference surface to the point. It has no physical meaning — water does not necessarily flow from a higher ellipsoidal height to a lower one — and it is exactly what the elevation factor needs, because the elevation factor is a purely geometric statement about two radii.

An orthometric height is the distance along the plumb line from the geoid, and it is what every levelling network in the world publishes because it is the height that governs which way water flows. It is the physically meaningful one and it is the wrong one here.

So the surveyor holding the physically meaningful number has to convert it, using a geoid model, into the physically meaningless one, in order to compute a horizontal distance correctly. That is a genuinely counter-intuitive requirement and it is why the mistake is common.

The geoid separation is a citation #

This collection decided among its essays on datums that a published spherical-harmonic geoid model counts as a citation rather than a measurement, for the same reason a coastline dataset does: the model has a truncation degree, so the separation it reports is partly a measurement of that choice and the reader cannot tell how much.

So the 48.5 metres above is a parameter of the figure rather than a computed quantity, and it is stated as such. What is computed is the mechanism — the ratio $N/R$, the resulting error on a stated line, and the comparison with the job’s tolerance — which is exactly as much as this collection’s own rules permit and is enough for the argument.

A reader with a geoid model for their own area substitutes their own separation and the arithmetic follows. A reader without one now knows how large the number has to be before it matters: about 130 metres of separation to reach 20 parts per million, which is most of the range the Earth’s geoid actually covers.

What happens when the chain is skipped #

The reductions are not optional in the sense that a job may decide against them. They are optional in the sense that a job may fail to apply them and produce coordinates anyway.

A survey that works entirely in ground distances and treats them as grid distances produces a set of coordinates whose internal geometry is perfect and whose scale is wrong by the combined factor — about 435 parts per million on this example. Every angle is right. Every ratio is right. Every closure closes.

That is the same similarity failure as a wrong foot, and it is invisible for the same reason: a traverse must close shows that a closed figure is blind to a uniform scale error by construction, because scaling a closed figure leaves it closed.

So the chain is not merely a set of corrections to apply carefully. It is a set of corrections whose omission cannot be detected from inside the job, which is why a specification names them explicitly rather than trusting a closure to reveal a problem.

What was computed, and how #

The chain, backwards. The grid distance is computed from the two points’ grid coordinates, and every step is then undone to recover what the instrument would have read — rather than forwards from an assumed tape reading, because the coordinates are the quantity that exists and the reading is the quantity being explained.

The line scale factor, by the integral rather than the endpoint value, using the machinery in the scale factor of a line.

The radius of curvature, as the Gaussian mean $\sqrt{MN}$ at the line’s mid-latitude, from the ellipsoid’s own constants.

The height mistake, as the difference between the elevation factor computed with the ellipsoidal height and with the orthometric one.

Two assertions, described above, the second of which was rewritten after the first version failed.

The chain in reverse, which is a different chain #

Setting out runs the same four steps backwards, and reversing a chain means reversing the order as well as the operations.

For the three multiplications that does not matter: multiplication commutes, so dividing by the combined factor in any order gives the same answer. Where it does matter is at the ends of the chain, where the corrections depend on quantities the other corrections have changed — the arc-to-chord correction depends on the grid positions and the convergence on the geodetic ones, so applying either to the output of the other applies it at the wrong point.

That non-commutation is measured in setting out runs the chain backwards, and it is small: about a millimetre of lateral offset at the far end of a fifty-kilometre line. Small, and real, and the reason the corrections have an order rather than a sum.

The same structure appeared one field along, when the seven parameters turned out not to invert by negation — scale and translation do not commute, and undoing them in the wrong order costs 1.3 centimetres. Too small to see, too large for a land registry, and caught only by requiring a round trip.

Where the model stops #

The slope angle is a single number, and real ground is not. A line over a ridge has a slope that changes sign, and the reduction is then an integral like the scale factor’s rather than a single cosine. Instruments handle this by measuring in short legs, which is a different subject and is why the number here is illustrative of the term’s size rather than of any particular job.

The line is a chord, not a geodesic. At 77 kilometres the difference between the chord through the ellipsoid and the geodesic across its surface is about 40 millimetres, which is comparable to the height mistake and is not in the chain above. A full reduction includes it; this one is stated as three terms plus a mistake because that is the comparison the essay is about.

And nothing here is about the instrument. Every real reduction begins with corrections for temperature, pressure, humidity and the instrument’s own additive constant, which between them are worth parts per million and are applied before any of this. They are omitted because they are properties of the equipment rather than of the coordinate system.

The generalisation #

The terms of a calculation should be ranked by what the answer is held to, not by their share of the answer. That is a general statement about error budgets and it is routinely violated in exactly the way this essay describes: the terms are listed in descending order of size, attention is allocated down the list, and the list stops where the terms stop looking big.

What makes the cartographic case sharp is that the tolerance and the total are five orders of magnitude apart. A 77-kilometre line closes to centimetres, and the reductions applied to it are hundreds of metres — so every term in the chain is enormous relative to the tolerance, and the ranking by share carries no information about the ranking by importance at all.

Who worked it out, and when #

The three reductions are old and were understood separately before they were understood together. Slope reduction is as old as measuring on hills. The reduction to sea level — as it was then called — dates from the eighteenth-century arc measurements, where it mattered because the baselines were measured on whatever ground was flat enough and the arc was defined on the reference surface. The grid reduction arrived with national grids in the twentieth century.

The height mistake could not exist until the 1980s, because before satellite positioning nobody had an ellipsoidal height and the reduction to “sea level” was performed with the only height available. The distinction became necessary at exactly the moment the two heights became separately obtainable, which is why the older textbooks are silent on a question the newer ones lead with.

A distinction that arrived with an instrument #

The height mistake being impossible before the 1980s is worth reading as more than a dating note, because it says something about how a subject’s errors change.

A mistake requires the two things it confuses to be separately available. Before satellite positioning there was one height — the levelled one — so the reduction used it and nothing was ambiguous. The reduction’s formula wanted an ellipsoidal height and the practitioner had an orthometric one, and the substitution was silent because there was no alternative to compare it against.

So the older practice was not making an error; it was making the only computation available, with a discrepancy absorbed into an accuracy budget that could not resolve it anyway. The formula and the practice agreed because nothing separated them.

The instrument created the distinction and the mistake at once. A receiver reports an ellipsoidal height directly, so a practitioner now holds both quantities, and choosing the wrong one is possible for the first time. That is why the newer textbooks lead with a question the older ones do not mention: the question did not exist for them.

Which is a general pattern in a measuring subject. A new instrument does not only make old quantities more precise; it splits quantities that were previously one thing, and every split creates a class of error that could not have been committed before. The epoch splitting a coordinate from its date, the geoid splitting a height from a distance, and the two heights here are three instances of the same event, and in each case the literature written before the split is silent rather than wrong.

Where this goes next #

The chain produces a coordinate. Whether the coordinate is right is checked by closing a figure, which turns out to be blind to the largest thing that can go wrong with it: a traverse must close.