The normal section is not the geodesic

Ask what the line between two points on the Earth is and there are three natural answers, all of them right about something, and no two of them the same curve.

An instrument at the first point sighted on the second swings in the plane containing its own vertical and the target. That plane cuts the ellipsoid in a curve — the normal section from A — and it is what an observed direction physically is.

The instrument at the second point sighted back swings in a different plane, because the two verticals are not coplanar unless the points share a meridian. The reciprocal observation traces a different curve.

And the geodesic, the shortest path, is neither. It runs between them.

Why the two sections differ at all #

The normal to an ellipsoid at a point does not pass through the centre. It passes through the axis, at a point that depends on the latitude, and two normals at different latitudes meet the axis at different places.

So the plane through A’s normal and B, and the plane through B’s normal and A, are different planes: each contains the two points but they tilt differently about the line joining them. On a sphere every normal passes through the centre, all such planes contain the centre, and the three curves collapse into one great circle — which is why nothing in the site’s spherical essays needs this distinction and everything ellipsoidal does.

The separation is therefore a direct measure of the flattening, and it vanishes with it. Measured on a sphere the two sections are $2\times10^{-9}$ metres apart, which is the arithmetic’s own noise.

The curve is cut, not stepped #

Both sections are computed exactly rather than integrated, by a change of variable worth stating because it removes a whole class of error.

Scaling the third coordinate by $a/b$ turns the ellipsoid into a sphere of radius $a$ and turns any plane into another plane. A plane’s intersection with a sphere is a circle, which can be parameterised in closed form, and undoing the scaling brings the circle back as the ellipse the section really is.

So the section’s points are exact. Only its length needs numerical work, and even that is a sum of chords rather than a differential equation.

The lengths needed a trick, and the first version failed on it #

Summing chords underestimates an arc, by $L^3/24R^2$ per chord. Over 3,026 kilometres in 400 segments that is 17 centimetres, and the quantity being measured — how much longer a normal section is than the geodesic — is 2.4 millimetres.

The first version of this measurement reported the normal sections as 17 centimetres shorter than the geodesic, which is geometrically impossible and read as a sign error.

The fix is Richardson extrapolation. The chord sum’s error falls as $1/m^2$, so evaluating at $m$ and $2m$ chords and taking $(4L_2 - L_1)/3$ cancels the leading term and leaves a residual below a millimetre. With that in place the excesses come out positive and small — 2.40 millimetres for the section from A and 0.71 for the section from B — and the geodesic is shortest, as its name requires.

That is the whole reason a measurement like this needs a control. “Shortest” is a claim that can be checked, the check failed, and the failure was in the measurement rather than in the geometry.

The normal is the object the whole distinction rests on #

Everything above follows from one fact about the ellipsoid: the normal at a point misses the centre.

The distance from the surface to the axis along the normal is $N(\varphi)$, and the distance to the centre along it is $N(1-e^2\sin^2\varphi)^{1/2}$ — a different number at every latitude. Two points at different latitudes therefore have normals meeting the axis at points separated by

$$N(\varphi_2)e^2\sin\varphi_2 - N(\varphi_1)e^2\sin\varphi_1$$

which is up to 43 kilometres between the equator and the pole. The two planes are tilted apart by an angle of that order divided by the distance between the points, and the curves they cut differ accordingly.

The separation is cubic in the length #

The two sections’ separation grows very fast, which is why it is invisible on a survey baseline and unignorable on a continental line.

A cubic law means the effect is genuinely absent at survey scale and genuinely present at national scale. At 50 kilometres it is 0.6 millimetres, which no instrument has ever resolved. At 200 it is 4 centimetres, which is a first-order network’s tolerance. At 800 it is 2.4 metres, which is a boundary dispute.

It depends on the azimuth, and vanishes along a meridian #

The separation is not a property of the length alone. Along a meridian it is exactly zero, because both points then lie in one plane containing both normals and the axis, and all three curves coincide.

The maximum is near 45°, which is the $\sin 2\alpha$ dependence the classical treatments give, and the residual 0.8 metres due east is because a geodesic launched due east does not stay due east — its azimuth changes along the way, which is Clairaut’s relation doing what it does.

It depends on the latitude too, falling from 10.4 metres at the equator to 0.4 metres at 75° for a 1,000-kilometre line on a 45° azimuth. That is a $\cos^2\varphi$ dependence, and it says the effect is largest exactly where a spherical treatment is otherwise safest.

That pairing is worth dwelling on, because it inverts the usual advice. The flattening’s effect on distance grows towards the poles — the meridian arc’s departure from a sphere’s is largest at high latitude, which is what the figure of the Earth was measured turns into an expedition to Lapland. Its effect on which curve is largest at the equator. A survey near the equator running lines at 45° is in the worst case for this quantity and the best case for the other, and no single rule of thumb covers both.

What a projection does with it #

A map is drawn from coordinates, so the three curves reach the page through whatever projection is in use, and the projection adds its own bending on top.

On a conformal projection a geodesic is drawn as a curve whose departure from the straight line between its endpoints is the arc-to-chord correction — third order in the length, like everything else here — and the two normal sections are drawn as two more curves a metre or so either side of it. At any scale a sheet is printed at, all three are inside one line width, so no map has ever shown the difference and none ever will.

That is the honest place to leave the cartographic consequence. The distinction matters to a computation and to a legal description, and it is invisible to a drawing. It belongs on this site because it is a fact about the body being mapped, and because the alternative — treating “the line between two points” as a single well-defined object — is exactly the kind of unexamined phrase what a coordinate refers to exists to take apart.

What is taught wrongly #

Three things, and the first is the one that appears in print most.

“The line between two points is the plane section.” It is not, it is two plane sections, and neither is the shortest path. A textbook figure showing “the normal section” with one curve is showing an object that does not exist.

“A reciprocal observation checks the first.” It checks the angle and not the curve. Two instruments sighting each other are looking along different curves on the ground, so a description of a boundary as “the line observed between A and B” is ambiguous by up to a metre over a national baseline — and which of the two curves is meant is a legal question that geometry cannot settle.

“The difference is negligible.” In length, yes — 2.4 millimetres over 3,026 kilometres is one part in $1.3\times10^9$, well below the accuracy of any distance measurement over that range. In position, no: 79 metres of lateral separation is not negligible for anything, and the two facts sit uncomfortably together. The wrong curve measures the right distance along the wrong ground, which is the sentence this essay exists for.

A fourth curve, which is the one most files contain #

There is a fourth answer to “the line between two points” and it is the commonest of all: the straight line in whatever plane the coordinates were stored in.

Putting the four in order of size settles what deserves attention. The stored straight line is out by kilometres; the spherical model is out by kilometres of distance; the two normal sections are out by tens of metres of position; and their lengths differ by millimetres. A treatment that worried about the last while writing the first is the ordinary state of the subject, and a straight segment is a claim about a plane is where the first is measured.

Which curve a survey actually uses #

None of the three, in general, and the reason is instructive.

Classical geodesy observed directions, which are normal sections, and computed on geodesics, because the geodesic is the curve with a closed-form-ish treatment and a symmetric relationship to both ends. The gap between what was observed and what was computed was handled by a correction with a name — the normal section to geodesic correction — applied to the observed azimuth, of order one third of the angle between the two sections at the observing end.

Modern practice observes coordinates rather than directions, so the question does not arise for the observation and does arise for everything computed afterwards: a bearing between two published coordinates is a geodesic azimuth, and an instrument set out on it will not be looking along a geodesic.

That is the same shape as setting out runs the chain backwards — a chain of corrections that has to be undone in the reverse order — and it is one more step of the chain that a spherical treatment silently omits.

The size of the correction says when it can be dropped, and the tolerance decides the model is the essay that turns every such correction into the line length at which it alone exceeds a stated tolerance. For this one that length is 400 kilometres at a decimetre and 800 at a metre — which puts it firmly outside ordinary survey work and firmly inside national control.

The check that makes the whole thing a measurement #

Four clauses, and the last is the one that keeps the machinery honest.

The two sections must differ, by metres over a long line, or the distinction is not real.

The geodesic must be shorter than both, which is what its name means and is a check on all three constructions at once: a sign error anywhere breaks it.

The separation must be cubic, fitted at 2.97 over the short lines, which turns an observation into a rule.

On a sphere all three must coincide, to $2\times10^{-9}$ metres. The whole effect is the flattening, so a routine that produced a separation on a sphere would be measuring its own discretisation and reporting it as geodesy. That refusal is why the ladder above can be trusted at 0.6 millimetres — a number smaller than most of the arithmetic’s own steps.

What it looks like as a route #

Drawing all three curves on a map of the region shows one line, because 79 metres over 3,026 kilometres is a fortieth of a millimetre at any printable scale. That is why the figure at the top of this essay plots the offsets rather than the curves.

Drawn on a world map at any printable scale the four curves are one line. The spherical model is the outlier by kilometres and the three ellipsoidal curves separate by metres, so a picture of them is a picture of a single stroke.

The site’s own convention follows from the measurement. Route figures draw the geodesic, because it is the shortest path and is symmetric in its two ends; the normal sections appear only where the distinction is the subject, as here. That is stated because it is a decision: a figure of an observed line would be a normal section, and no figure on this site draws one.

The same defect, one dimension down #

There is a familiar version of this on a plane, and holding the two together makes the ellipsoidal case less exotic.

Two points on a grid have a straight line between them, and a straight line on the grid is not a straight line on the ground: grid north is not north measures the angle, and the arc-to-chord correction measures the offset. The pattern is identical — an observation follows one curve, a computation follows another, and the two are reconciled by a correction that is third order in the length.

What differs is only where the curvature comes from. On the grid it is the projection’s; here it is the ellipsoid’s own asymmetry between two normals. Both produce a curve of order $s^3$, both vanish along one special direction, and both are handled in practice by a correction applied to an angle rather than by using the right curve.

A straight line in a treaty is three curves #

The three answers matter most where a document says straight line and means one of them without saying which, and boundary description is where that happens.

The phrase is genuinely ambiguous on an ellipsoid. A border defined as running straight between two monuments could follow the geodesic, either of the two normal sections, or the curve a chord through the body traces on the surface — and older instruments frequently do not say. The separations are metres over a boundary of a few hundred kilometres, growing as the cube of the length and vanishing only along a meridian.

Metres of boundary are not a rounding error. They are a strip of land with an area, they may hold a wellhead or a jetty or a road, and they are exactly the kind of thing that gets litigated a century after the document was drafted — at which point the question is not what the geometry is but what the drafters meant, and the drafters may not have known there was a choice.

Modern practice names the curve. Contemporary boundary treaties and maritime delimitations specify the geodesic explicitly, along with the ellipsoid it is computed on, and the specification is now standard enough that its absence in an older document is itself informative.

It is worth noting which way the ambiguity resolves in practice, too: a court asked to interpret a silent document generally reaches for the parties’ contemporaneous maps rather than for the geometry, so the curve that governs is whichever one somebody happened to plot.

The general shape is the one this whole ladder keeps producing. A phrase that is unambiguous on a plane acquires several meanings on a curved body, the meanings differ by a quantity that is small in relative terms and material in absolute ones, and the difference is invisible until somebody computes two of them.

Where this ladder goes #

The ellipsoid anchor has taken the body from “the Earth is not a sphere” through the two latitudes that differ by 11.5 arcminutes, the datum shifts that dwarf projection errors, the series that computes a transverse Mercator, the zone system built on it, and the grid north that is not north.

This adds the last of the ellipsoid’s own geometry the site had left out: what a straight line is on it, which turns out to be three different questions with three different answers whose disagreement is a fifth-order quantity in the flattening and a first-order quantity in a boundary dispute.

What remains is the observation itself — what an instrument’s vertical actually is, as opposed to the ellipsoid’s normal — and that is the deflection of the vertical, which this site holds as a relation rather than a value and will not compute until it has a defensible way to do so without a gravity field it does not own.