A line of position is Newton's method, but only on a conformal chart

Every essay about routes starts from a ship that knows where it is. Why Mercator exists is about holding a course from a known position; the shortest route is not straight is about the route between two. Neither says how the position was known, and for most of the history of ocean navigation it was not known at all until somebody measured the height of a star.

That measurement gives a circle rather than a point, and the circle is far too large to draw. The method navigators used for a century and a half to turn two such circles into a position replaces each one with a straight line on the chart, and it works so well that it is easy to miss that it depends on the chart. On one kind of chart it is an exact numerical method with a convergence rate. On another it is wrong at first order, by an amount with a closed form.

An altitude is a circle #

A star’s altitude is its angle above the horizon. At any instant there is one point on the Earth directly beneath the star, its geographical position, and from there the star is straight overhead. From a place a quarter of the way round the Earth it is on the horizon. In between, the altitude falls off with angular distance from the geographical position, degree for degree: the altitude is ninety degrees less that distance.

So an observed altitude says exactly one thing: the observer is at a stated angular distance from a stated point. Every place at that distance forms a circle on the sphere, the circle of equal altitude, and the observer is somewhere on it. A circle of a distance is not a circle is about what a map does to exactly this kind of set, and these ones are large: an altitude of 45 degrees puts the observer on a circle 5,000 kilometres in radius.

A second body gives a second circle, and the two cross twice. One crossing is the ship. The other, for these two bodies, is 5,342 kilometres away, and no navigator is in doubt about which is which. The problem is entirely the size: the part of each circle that matters is a few tens of kilometres long, and the whole circle would not fit on any chart a ship carries.

The line that stands in for it #

The intercept method, published by Marcq Saint-Hilaire in 1875, never draws the circles. The navigator starts from an assumed position — a dead-reckoning estimate, or a convenient round-numbered point near it — and computes two things for each body: the altitude it would have if the ship were there, and its azimuth from there. The difference between the observed and computed altitudes, in minutes of arc, is a distance in nautical miles.

On the chart, the navigator draws the azimuth from the assumed position, marks off that distance along it — towards the body if the observed altitude is greater, away if less — and draws a straight line through the mark at right angles to the azimuth. That line is the line of position. The fix is where two such lines cross.

The construction is a first-order approximation made twice. Near the assumed position, the circle of equal altitude is nearly straight and runs perpendicular to the direction of its centre, which is the body’s azimuth. And the altitude changes by one minute of arc for every nautical mile moved along that azimuth. Both facts are exact for infinitesimal distances and approximate for the tens of kilometres by which an assumed position is usually wrong. There is a third approximation inside the drawing itself: on Mercator a line laid off at a constant compass angle is a rhumb line, not the great circle along which the altitude actually changes fastest. The line drawn straight on the page is a route prices that difference over whole crossings. Over an intercept of 111 kilometres at 50° north the two end up as much as 1.15 kilometres apart — but that separation lies along the line of position, which leaves the line where it was, and the part across it, the only part that moves the line, is at most 16 metres. That is inside the second-order remainder measured below.

In the hero figure the assumed position is 150 kilometres from the ship, which is a large error for a navigator but a useful one for seeing the construction. The two intercepts are 12.2 and 111 kilometres. The faint circles curve away from the straight lines, and the lines cross 3.28 kilometres from the ship.

On Mercator, a fix is one step of Newton’s method #

The shape of that 3.28 kilometres is the first thing worth measuring. Moving the assumed position further from the ship and closer to it, in every direction, shows how the fix error depends on the error it started from.

On Mercator the fix error falls as the square of the assumed position’s error. An assumed position 10 kilometres out gives a fix 17 metres out; 40 kilometres out, 278 metres; 320 kilometres out, 18.2 kilometres. On log axes the fitted slope is 2.005.

That is the signature of Newton’s method, and the intercept method is Newton’s method, exactly. A fix solves two equations — the altitude of each body as a function of position equals the observed altitude — for two unknowns. Newton’s method replaces each equation by its linearisation at the current estimate and solves the linear pair. The linearisation of an altitude function at a point has its gradient along the body’s azimuth, with a magnitude of one minute of arc per nautical mile, and its level set is the straight line perpendicular to that gradient. That is the line of position. Two lines crossing is two linear equations solved. The residual error of one Newton step is second order in the starting error, and the slope of 2.005 is that. It is the same square law flying a curve in straight legs found for a great circle replaced by chords, and for the same reason: a straight piece standing in for a curve departs from it by the curve’s curvature times the square of the length it covers.

The step is only Newton’s if the chart keeps the linearisation intact, and that is where Mercator comes in. The navigator lays off an azimuth as a compass angle and a perpendicular as a right angle, and both are the true directions on the ground only if the chart preserves angles at the assumed position. Mercator does, everywhere. A plotting sheet whose longitude scale has been shrunk by the cosine of the assumed latitude does too, at the assumed position — which is all a single step needs — and its slope is 2.003.

Run again from its own fix #

Newton’s method is usually iterated, and so is the intercept method: a navigator who distrusts a fix can reduce the same two sights again from the fix as the new assumed position.

On Mercator, from 300 kilometres out, the first fix is 13.2 kilometres from the ship, the second 24 metres, and the third closer than the arithmetic can measure. Each step roughly squares the error in units of the circle’s size, which is quadratic convergence; the corrected plotting sheet goes 300 kilometres, 16.5, 41 metres, and then the same.

This is why a navigator almost never needs to re-run a fix. From a dead-reckoning position thirty kilometres out in any direction, the first fix on Mercator is within 156 metres — far inside the error of the sextant, whose minute of arc is a nautical mile. The intercept method is not an approximation that happens to be good enough. It is one step of a method whose second step is never needed.

Corrected at one latitude, and nowhere else #

The corrected plotting sheet is the navigator’s usual instrument, a blank grid whose meridians are spaced by the cosine of the latitude being worked at. It keeps angles exactly at that latitude and nowhere else, so it is second order like Mercator, with a slightly larger constant: thirty kilometres out in the worst direction its fix is 176 metres from the ship against Mercator’s 156, about twelve per cent more. The difference is the sheet’s own departure from conformality across the few tens of kilometres the construction spans, and it is second order too.

The same step in every satellite receiver #

The method did not end with the sextant. A satellite receiver fixes its position from ranges rather than altitudes — each range puts it on a sphere about a satellite, as each altitude puts a ship on a circle about a star — and it solves for the crossing the same way: linearise every range about an assumed position, solve the linear system, and repeat from the answer. That is Newton’s method again, with a least-squares solve when there are more ranges than unknowns, and it converges quadratically for the same reason. The receiver never draws a chart, so the question of which chart keeps the step exact does not arise; its linear algebra is done in the three-dimensional coordinates where the gradient of a range is simply the direction to the satellite.

A sheet that forgets the latitude #

The third curve in both figures is a sheet that does not keep angles: a plotting sheet whose degree of longitude is drawn the same length as its degree of latitude. It is a plate carrée about the assumed position — the plate carrée, the projection nobody chooses, in its most local form — and a navigator who laid down a blank grid of squares and forgot the correction would be plotting on it.

On that sheet the fix error falls only as the first power of the assumed position’s error. At 10 kilometres out it is 3.57 kilometres, more than two hundred times Mercator’s. At 320 kilometres it is 115. And re-running does not rescue it: from 300 kilometres out the errors go 81.8, 28.8, 10.3, 3.67, 1.31 kilometres, each a fixed 0.357 of the one before, and after six re-runs the fix is still 468 metres from the ship. That is linear convergence, and a navigator doing this would conclude, correctly, that the method was converging and, wrongly, that it was converging on the right answer at a normal rate.

The error has a direction.

An assumed position that is wrong only in latitude gives a fix as good as Mercator’s: 0.0006 of a kilometre per kilometre of error, which is just the second-order term. One that is wrong only in longitude gives a fix wrong by 0.3572 of a kilometre per kilometre. In between, the share follows the sine of the error’s direction from north exactly, and the dashed curve $(1 - \cos 50°)\,|\sin\beta|$ passes through every measured point.

One less the cosine of the latitude #

The number 0.3572 is $1 - \cos 50°$, and the reason is short.

The navigator lays off every intercept in kilometres, at the sheet’s north–south scale. On a sheet whose longitude is not corrected, a kilometre drawn east is read back as a kilometre of longitude-degrees at the equator’s scale, which at latitude $\varphi$ is only $\cos\varphi$ of a kilometre on the ground. So when the lines of position cross and the crossing is read off as a latitude and longitude, the east–west part of the correction is applied at $\cos\varphi$ of its proper size. A fix from an assumed position wrong by $e$ kilometres east recovers $e\cos\varphi$ of it and keeps $e(1 - \cos\varphi)$. A north–south error is read at the right scale and fully recovered.

The same number governs both the first fix and every re-run, because each re-run starts from a fix whose east–west error is the part the previous one kept. At 30° the uncorrected sheet keeps 0.1339 of it each time; at 60°, exactly half; at 80°, 0.8263, so each re-run there removes less than a fifth of what is left. At the equator it keeps nothing — the sheet is Mercator there — and its error falls as the square again, with a fitted slope of 2.02.

The two regimes are cleanly separated. On any chart that preserves angles at the assumed position, the fix error is second order. On a chart that does not, it is first order, with a constant fixed by how far the chart departs from preserving them. A crossing bends by a law only a conformal chart can show found a law of angles that a protractor can check on Mercator and cannot on a plate carrée; this is the same distinction arriving as a convergence rate.

Why the correction is the whole method #

It is tempting to treat the latitude correction on a plotting sheet as a refinement — a cosine applied to a scale so that the drawing looks right. The measurement says it is the difference between a second-order method and a first-order one.

A navigator at 50° north with an assumed position 20 kilometres wrong in longitude gets a fix 61 metres out on a corrected sheet and 7.1 kilometres out on an uncorrected one. The first is an order of magnitude inside a sextant’s accuracy. The second is a position that would put a ship on the wrong side of a harbour entrance, from a construction every step of which was performed correctly.

That is also why the error is hard to notice. The uncorrected construction produces a fix, the fix is close to the assumed position, and re-running it moves it steadily towards the truth. Nothing about the drawing reveals that it is drawn in the wrong geometry. Conformal does not mean the angles are right is about the limits of what a conformal chart preserves; this is about the case where preserving angles at one point is exactly and only what a method needs, and a chart that does not is wrong in a way no inspection of the plot can see.

What each number was checked against #

The truth is the ship. Every sight is computed from the ship’s true position, so the altitude and azimuth of each body are exact, and every fix error is a distance on the sphere from a known point.

An assumed position at the ship must give the ship. Both intercepts are then zero and every line passes through the assumed position, on every chart. On Mercator, on the corrected sheet and on the uncorrected one the error is zero to the arithmetic, which is the refusal the whole measurement rests on: a method that moved a correct position would be measuring its own arithmetic.

The orders are fitted, and bounded. Mercator’s and the corrected sheet’s slopes must lie within a tenth of two, and the uncorrected sheet’s within a tenth of one; they are 2.005, 2.003 and 1.002.

The closed form must be met at three latitudes. At 30°, 50° and 70° the uncorrected sheet’s first-order share and its per-step factor must both equal $1 - \cos\varphi$ to one per cent. They agree to four decimal places.

And at the equator the uncorrected sheet must be second order again, because there it is Mercator. Its slope there is 2.02.

What the sphere and the two bodies leave out #

The Earth is a sphere. An altitude is measured from the local vertical, and on the ellipsoid the altitude formula stays exact when fed the geodetic latitude, because the vertical is the ellipsoid’s normal. What the sphere gets wrong is the conversion of an intercept into a distance: a minute of meridian arc is 1,842.9 metres at the equator and 1,861.6 at the pole, so a nautical mile is a minute of arc only to half a per cent. And at sea the true vertical departs from the ellipsoid’s normal by a few seconds of arc. A bearing on a sphere is decided by its latitude, not its radius prices the azimuth side of that substitution; neither changes an order of convergence.

The sights are exact. A real altitude carries a sextant’s error, refraction and the height of the eye, and a fix from two sights with independent errors has an uncertainty of its own that the construction does not remove. The fix errors here are what the construction adds to perfect sights, and they are the part a navigator can control by choosing the chart.

Two bodies, crossing at 51°. The lines of position cross at the same angle as the two azimuths, and a fix from lines that cross at a shallow angle amplifies every error in either. Two bodies nearly in line would make every number above larger, and the orders the same.

Still open: three lines, and the triangle they make #

Navigators do not usually stop at two sights. A third body gives a third line, and three lines of position almost never pass through one point: they form a small triangle, the cocked hat, and the navigator has to decide where in or near it the ship is.

Everything measured here says that on a conformal chart the three lines are, to second order, the true tangents, so the triangle is made by the sights’ own errors rather than by the construction. That changes what the triangle can mean. Whether the ship lies inside it as often as it looks as if it should, whether a common error in all three altitudes — a sextant misread by the same amount each time — moves the best estimate somewhere the triangle does not contain, and whether the uncorrected sheet’s first-order error enlarges the triangle or merely shifts it, are questions two lines of position cannot ask.