The shortest route is not at sea level

Every rung of this ladder routes on a surface. The shortest path between two places is a great circle on the sphere and a geodesic on the ellipsoid; the gnomonic draws it straight; flying it takes straight legs; the ellipsoid’s cut locus is where it stops being unique.

Nothing is ever flown on a surface. An aircraft cruises at about eleven kilometres, a satellite at hundreds, and both take the shortest path available at their own height rather than the shortest path along the ground below.

The obvious correction is a factor. A great circle at height h is longer than the one at the surface by 1 + h/R, which on a transatlantic crossing is about ten kilometres, and that is the whole story — on a sphere.

Because the offset of a sphere is a sphere #

A sphere of radius R, offset outwards by h everywhere along its own normal, is a sphere of radius R + h. So the geodesic up there is a great circle of the larger sphere, its ground track is the great circle of the smaller one, and the two lie exactly on top of each other. The only difference is the length, and the length is the naive factor exactly.

And the offset of an ellipsoid is not an ellipsoid #

Offset an ellipsoid outwards by h along its own normal and the surface that results has principal radii of curvature M(φ) + h and N(φ) + h. That is the standard result and it is easy to check: a normal section’s curvature is the reciprocal of its radius, and moving out along the normal by h increases that radius by exactly h.

Those two functions do not belong to any ellipsoid. An ellipsoid with semi-axes a + h and b + h has a smaller flattening, (a − b)/(a + h), and its radii of curvature are not M + h and N + h. So the offset surface is a surface of revolution with a shape no ellipsoid has, and the geodesic equations on it have different coefficients from the ones on the ellipsoid below.

Which is enough. A geodesic on a surface of revolution obeys Clairaut’s relation — the distance from the axis times the sine of the azimuth is constant along the path — and the distance from the axis is (N + h) cos φ up there against N cos φ down here. Two different relations, two different curves, and the same two endpoints.

Eleven metres, and what they are made of #

The number is small and it is not nothing.

It is larger than the geodesic algorithm’s own error. Vincenty’s method and the integrated geodesic agree to under half a metre over the same route, and modern algorithms are exact to nanometres. Eleven metres is well above the floor of the arithmetic.

It is proportional to the height. 1.18 metres at 1.1 kilometres, 11.8 at eleven, 116 at a hundred and ten, and 404 at four hundred — the last one falling below the line as the linearisation runs out. A quantity proportional to h and vanishing at h = 0 is the offset surface’s doing and cannot be anything else.

It is a lateral departure, not a length. The two routes leave and arrive at exactly the same places. What differs is the ground they pass over in the middle.

The length is not the naive factor either #

The lateral departure is the surprising half. The length has a second, smaller finding in it.

The mechanism is one line. The extra length is h times the integral of the normal curvature along the path, and on an ellipsoid the normal curvature depends on the latitude and on the azimuth — the curvature in the meridian is 1/M and across it is 1/N, and a general azimuth is Euler’s combination of the two. A route that spends its length at high latitude accumulates a different total from one along the equator, and the naive s/R assumes they are the same.

The discrepancy is a few tens of metres out of sixteen kilometres. It is a measurement of the flattening rather than of the height, and it is exactly the same size on every route at every altitude — the ratio is 0.998655 for London–Tokyo at one kilometre and at four hundred.

The same question one level down #

There is a rung on the wrong ladder that asks a question of exactly this shape and it is worth reading beside this one.

The normal section is not the geodesic finds that the curve an instrument sights along — the intersection of the ellipsoid with the plane containing the two points and one of their normals — is not the shortest route between them, and that the two normal sections between a pair of points are not even the same curve. The departure is centimetres over a hundred kilometres and it is real.

The structure is identical. In both cases a curve everybody calls “the line between two points” turns out to be several curves; in both cases the difference is proportional to the flattening and vanishes on a sphere; and in both cases the size is far below any operational tolerance and far above the arithmetic’s floor.

What is different is where the ambiguity comes from. There it is the instrument — two normals, two planes, two sections. Here it is the height — one family of surfaces, one geodesic on each. Between them they say that “the shortest route between London and Tokyo” needs two things named before it is a curve: the surface, and which of the several natural curves on it is meant.

What the number is proportional to #

Three factors decide the departure and all three are visible in the measurements above.

The height, linearly. A fitted slope of 1.00 over three decades, which is what a first-order perturbation of the geodesic equations gives: the coefficients change by h/N and the track responds in proportion.

The flattening. Zero on a sphere to 5 × 10⁻¹⁰ metres, so the whole effect is carried by the one number that separates the ellipsoid from the sphere. Halving the flattening would halve the departure.

And the route’s latitude range. The coefficients differ between the two surfaces by an amount that depends on how much N varies along the way, so a route along a parallel or along a meridian has less to work with than one that crosses many latitudes obliquely.

That third one is why the length discrepancy in the table changes sign between the Tokyo route and the Sydney route while the lateral departure does not — the two quantities depend on different integrals of the same varying curvature, and only one of them can cancel.

What the ladder had already assumed #

Three earlier rungs quietly answer for a height of zero and are worth re-reading with that stated.

The second rung carries the same silent assumption one level further in, because a length quoted for a rhumb line is a length along a surface nobody named either.

The pattern across all three is that the surface was never a variable. It was the ellipsoid, because that is what a coordinate refers to, and the question of which ellipsoid-shaped surface — the one at sea level or the one a trajectory is actually on — did not arise because nobody was asking about a curve in the air.

The reduction, run backwards #

There is a standard geodetic operation that is exactly the inverse of this and it is worth naming, because it is where the arithmetic already lives.

A measured distance between two ground stations is reduced to the ellipsoid before it is used, and the reduction divides out the station heights: a line measured at 2,000 metres is longer than the corresponding ellipsoidal chord by about 2,000/6,371,000 of its length, three parts in ten thousand, and every survey does this without thinking about it. What a tape measures prices it and finds the geoid separation entering the same correction.

That reduction handles the length correctly and says nothing about the track, because a survey line is short enough for the two to be the same curve to well inside any tolerance. Over a hundred-kilometre line the lateral departure this essay measures is under a tenth of a millimetre.

So the operation exists, it is old, and its assumptions are stated for the case it was built for. What this rung adds is what happens when the same reduction is applied over ten thousand kilometres instead of over ten: the length correction acquires a route dependence of a tenth of a per cent, and the track correction, which was zero, becomes eleven metres.

Where this matters, and where it does not #

Not for navigation. Eleven metres of lateral departure over nine thousand kilometres is far inside every tolerance an aircraft operates to, and the flight plan’s own waypoints are stated to a hundredth of a minute of arc, which is eighteen metres. Flying a curve in straight legs prices the departure from a few great-circle legs at kilometres; this is three orders of magnitude below that.

Possibly for a satellite. At four hundred kilometres the departure is four hundred metres, and an orbit’s ground track is computed to much better than that. The relevance is limited because an orbit is not a geodesic of any surface — it is a trajectory in a gravity field — but a ground track computed by projecting an orbital arc downwards uses exactly the geometry in this essay, and the projection is along the normal to which surface.

And for the statement, which is what a ladder is for. Every result on this ladder is about the shortest route on a surface, and this rung establishes that the surface has to be named. “The shortest route from London to Tokyo” is not one curve; it is a family of curves parameterised by the height it is asked at, they differ by metres, and every previous rung has been quietly answering for h = 0.

Where the model stops #

The offset surface is not where an aircraft is. An aircraft holds a barometric altitude, which is a surface of constant pressure, which is neither the ellipsoid offset nor the geoid offset and wanders by tens of metres with the weather. The eleven metres here is the answer to a clean geometric question and not to an operational one.

The shooting solve is Newton on a numerical integrator. Two unknowns, the departure azimuth and the distance, iterated until the arrival matches to 10⁻¹³ radians. Its own error is bounded by the integrator’s, and the sphere control at 5 × 10⁻¹⁰ metres is the measurement of that bound — which is the reason the control is run at all.

Only one route is drawn. The lateral departure depends on the route’s latitudes and azimuths in a way this essay does not characterise: London to Sydney behaves differently enough to flip the sign of the length discrepancy, and a route along the equator or along a meridian would give zero departure by symmetry. What is established is that the effect exists, is proportional to h, and vanishes on a sphere.

And the height is constant. A real trajectory climbs and descends, so its shortest path is a problem in a three-dimensional medium rather than on a surface at all — which has a different answer again and is not a geodesic problem.

Who found it, and when #

The geometry is nineteenth-century. Offset surfaces, parallel surfaces, and the fact that the radii of curvature simply add are in every classical differential geometry text, and the flattening of the offset ellipsoid being smaller than the original’s is an exercise.

The geodetic literature knows the length correction well: reducing an observed distance from the ground to the ellipsoid is one of the standard reductions, it accounts for the station’s height through exactly the 1 + h/R factor, and the better treatments use the normal curvature rather than a mean radius — which is the second finding above, arriving from the direction of a survey rather than of a flight.

What does not appear to be stated anywhere is the first finding: that the track moves, not just the length. The reason is probably that the two communities who would care have no need of it. Geodesy reduces to the ellipsoid and works there; aviation flies to waypoints and does not care what curve joins them. The question only becomes visible if somebody asks what the shortest route at altitude is, as a curve, which is a question with no operational customer.

The finding justifies the practice it says nobody needs #

The observation that the question has no operational customer is right and it is not the end of the argument, because the practice it describes is making a silent assumption that this rung has now checked.

A flight management system computes its legs on the ellipsoid. The great circle between two waypoints is a sea-level geodesic, computed with sea-level constants, and the aircraft flies it at eleven kilometres up. Nobody in that pipeline asks what the geodesic at altitude would be; the sea-level curve is simply used.

So the practice does depend on the answer, it just never asked for it. The question is the track at altitude the same curve as the track at sea level is one the system has answered implicitly, in the affirmative, by construction — and until it is computed, the affirmative is an assumption rather than a result.

The measurement says the assumption is sound. Eleven metres of track displacement over a long crossing is beneath every tolerance in the operation: beneath the navigation performance the airspace requires, beneath the width of the corridor, beneath the accuracy of the wind model that dominates the actual routing. The practice is correct and now has a number behind it.

Which is what makes the rung worth having rather than a curiosity. A collection like this has two kinds of finding: the ones that overturn a practice, and the ones that convert an unexamined habit into a checked one. The second is less exciting and more common, and the checking is the same work either way — the result is only known to be small after it has been computed.

It also gives the practice a place to record what it is relying on, which it currently does not have. A flight management system’s documentation states that legs are great circles on a named ellipsoid; it does not state that the aircraft flies them ten kilometres above that ellipsoid and that the difference has been judged negligible. The judgement is correct and unwritten, which is the same shape as every other unstated convention in this collection.

And the number is the useful part rather than the verdict. Eleven metres at eleven kilometres is a scaling anybody can apply: the displacement grows with the height, so a practice sound for an airliner is sound by a wider margin for a helicopter and would need re-examining for anything an order of magnitude higher — which is exactly the sort of thing a stated number supports and a verdict does not.

A judgement nobody wrote down is one nobody can revisit when the conditions change.

Where the ladder goes next #

The ladder now has routes on a sphere, on an ellipsoid, at altitude, in wind, round obstacles, and past the cut locus. The assumption it has never examined is the one this rung has just made explicit: that a route is a curve on a surface.

A real trajectory is a curve in space. Its length is not the length of its ground track, its shortest form is not a geodesic of anything, and the surface it is reduced to is a modelling choice with a cost — which this rung has measured for one particular choice and one particular height, and which nothing on this ladder has asked in general.