A straight segment is a claim about a plane

A line in a file is two points. Everything between them is supplied by whatever draws it, and what every renderer supplies is a straight line in the plane the coordinates are in.

That is a claim about the ground, and it is checkable.

The measurement is on the ground, not on the page #

The obvious way to measure this is on the picture: how far apart are the two curves in pixels. That number depends on the scale and says nothing about the Earth.

So the measurement is inverted instead. The drawn segment is sampled in the projected plane, each sample is carried back through the inverse projection to a latitude and longitude, and its cross-track distance to the great circle between the endpoints is computed — the standard geodesic quantity, and one this site already had for flying a curve in straight legs.

That gives a curve of metres against position along the line, which is what the lower panel of every figure here is. For the transatlantic case it peaks at 718 kilometres, 49 per cent of the way across — near the middle, as a chord against an arc must be, and the assertion demands it fall between 30 and 70 per cent so that a measurement drifting towards an endpoint would be caught.

How large it gets #

Three cases, all in the plane a web map’s coordinates live in:

The last one is not a small correction to a route; it is a different route. A straight line drawn between London and Tokyo on a web map passes over central Asia; the geodesic passes near the pole, which is the great-circle vertex and the reason polar flights exist.

The first one is the one worth carrying, though. Eighty-one metres over sixty kilometres is invisible on any picture and is thirty times the tolerance a survey chain is held to. A line that is a decoration on a map is a fault line in a dataset.

The rule, fitted rather than assumed #

A chord departs from an arc of curvature $\kappa$ by about $\kappa L^2/8$, so the departure should be quadratic in the length. That is a prediction and the site’s habit is to fit it rather than to cite it.

Six east–west segments at 45° north, from 50 to 1,600 kilometres, give worst departures of 49, 196, 784, 3,138, 12,566 and 50,462 metres — and a least-squares fit on the logarithms gives an exponent of 2.0012.

Two follows from the geometry, so the number itself is unsurprising. What the fit tests is whether the measurement is picking up the curvature and only the curvature: a projection’s own distortion or a sampling artefact would show as an exponent away from two, and the assertion refuses anything outside 1.9 to 2.1.

The control that makes it about the projection #

There is a projection on which every great circle is a straight line: the gnomonic, which is the whole content of the gnomonic companion and the reason it is carried beside Mercator on a navigator’s chart.

A stored segment between two points of a geodesic, drawn straight in the gnomonic plane, departs from the ground by $2.3\times10^{-7}$ metres — a fifth of a micron, which is the sampling and the arithmetic and nothing else.

That is the control that turns the rest of this essay from an observation into a measurement. The departure is not an inevitable consequence of drawing lines; it is a property of the plane the line was stored in, and there exists a plane in which it vanishes. The 718 kilometres belongs to Mercator, not to straightness.

What a tolerance buys #

The inverse question is the one an operation actually has: given that the drawn line must not stray more than so far from the ground, into how many pieces must it be cut?

Since the departure is quadratic, halving a piece quarters its error, so the vertex count goes as the reciprocal square root of the tolerance. Measured on a 4,605-kilometre line at 45° north:

A hundredfold tighter tolerance costs sixteen times the vertices rather than a hundred, which is the whole reason a metre-accurate world map is affordable at all. The assertion checks the direction and the magnitude: more vertices for a tighter tolerance, and fewer than the tolerance ratio would suggest.

Densifying along the wrong thing #

There are two ways to add vertices and only one of them helps, which is a distinction worth making because both are called densification.

Along the geodesic. Put the new vertices on the route the line is meant to follow. The stored line then approximates the ground to whatever tolerance was asked for, and — this is the part that matters — it does so in every plane, because the vertices are points of the Earth and every projection maps them to points of the picture. The chords between them differ slightly from plane to plane, at the residual level the table above quantifies.

Along the plane’s straight line. Put the new vertices on the drawn segment, evenly in projected coordinates. The picture is unchanged, the file is larger, and the line’s relationship to the ground is exactly as wrong as it was — it now documents the wrong route at high resolution instead of implying it at low.

The second is not a straw man: it is what happens when a line is densified after being reprojected, by a routine working in the target plane. The site’s own machinery does the first, and the difference between them is the difference between adding information and adding vertices.

The same line at three scales, which is where the intuition fails #

The three cases in the table span a factor of 160 in length and a factor of nine thousand in error, and holding both in mind at once is what the quadratic law is for.

At 60 kilometres the departure is 81 metres, which is 0.14 per cent of the line’s length. Nobody would call that a different route, and a survey would call it a blunder.

At 600 kilometres the same law gives about 8 kilometres, or 1.4 per cent. A courier planning a route would notice; a map reader would not.

At 6,000 kilometres it is 718 kilometres, 13 per cent, and the two lines are not comparable objects.

The percentages are the useful form: the relative error grows linearly with the length, because the departure is quadratic and the length divides once. So the question is a straight segment good enough has a clean answer — it is good to a stated fraction of its own length for lines up to a computable size, and the size follows from the fraction by one square root.

For a tenth of a per cent that size is about 45 kilometres, which is roughly the extent of a survey job and roughly the extent of a national grid zone’s usable width. That coincidence is not a coincidence: both numbers come from the same curvature, which is how small is flat enough inverted for a different tolerance.

What the rhumb line costs in distance, computed #

The essay’s last section notes that a ship steering the stored segment arrives some extra distance later and leaves the number out. It is worth having, because it behaves quite differently from the cross-track error.

Computing the loxodromic distance directly — the length of a constant-bearing course on the ellipsoid’s spherical stand-in — against the geodesic between the same endpoints:

Sixteen hundred and forty-four kilometres between London and Tokyo, on a route that is 9,559 long. That is not a correction; it is five hours of flying.

The two errors scale differently and the difference is the useful part. The cross-track departure is second order in the length and the extra distance is third, so the ratio between them grows: at sixty kilometres the line is 0.14 per cent off-track and 0.02 per cent long, and at nine thousand it is 31 per cent off-track and 17 per cent long. A short segment is therefore nearly the right length while being in visibly the wrong place, and a long one is wrong in both.

The leading-order relation between them is one line. If the bow reaches a maximum cross-track s over a length L, the extra path length is about (8/3)s²/L — which gives 247 kilometres for the transatlantic case against a measured 207, and 2,446 for the transpacific one against 1,644. Twenty to fifty per cent high, because the relation is derived for a parabolic bow and a rhumb line’s departure is not parabolic once it is a third of the route’s own length.

That over-estimate is worth keeping rather than refining. It is a bound on the extra distance computable from the cross-track figure alone, and the cross-track figure is the one this essay’s machinery already reports for any pair of endpoints in any plane. A planner with a departure profile and no route-length calculation has an upper bound on the distance cost for the price of one multiplication.

And the ordering it establishes is the practical one. A stored segment is off-track long before it is long, so the failure a reader would notice — a line drawn over the wrong continent — arrives at a shorter length than the failure a fuel calculation would notice. Between them there is a range of lengths, roughly a few hundred to a couple of thousand kilometres, where the picture is visibly wrong and every number computed from it is nearly right, which is the worst possible arrangement for anybody deciding whether to densify.

What it does to an intersection #

A departure of this size changes topological answers, and that is the consequence with the sharpest edge.

Two segments that cross in one plane need not cross on the ground, and two that cross on the ground need not cross in the plane. The crossing point, where there is one, is at a different place in each plane. So do these two lines meet is a question with a coordinate system in it, and a dataset assembled from two sources projected differently can contain a junction in one and a near-miss in the other.

For short lines the effect is small — the 81-metre departure over sixty kilometres bounds how far a crossing can move — and for long ones it is decisive. The general statement is the field’s recurring one, arriving now at a predicate rather than a measurement: nearest is a question about the metric shows the same for a ranking, and the two together are the reason an operation’s plane belongs in its specification.

What the departure does to a length #

A line that is not where it claims is also not the length it claims, and the two errors are related but not equal.

The chord of an arc is shorter than the arc, by $L^3\kappa^2/24$ to leading order — third order rather than second, so it grows faster with length and starts smaller. For the transatlantic segment the drawn route is longer than the geodesic rather than shorter, because it is a rhumb line rather than a chord and the extra cross-track distance has to be paid for.

The practical consequence is that a length measured off a stored line is a length of the wrong curve, before any projection scale factor is applied. Two corrections, of different origins, applied to the same number:

• the shape correction, because the vertices join along the wrong path — this essay;
• the scale correction, because the plane’s own scale factor is not one — the scale factor of a line, which integrates it along the route rather than sampling the ends.

For a densified line the first vanishes and the second remains. For a two-point line at continental scale the first dominates by orders of magnitude, and no amount of care with the second recovers anything.

Which plane a file should be stored in #

The essay’s practical question, and the answer is the field’s answer everywhere: it depends on what will be asked.

Stored in geographic coordinates, a segment is straight in the equirectangular plane, which is nobody’s route and is at least a plane every program can read. Stored in a projected system, the segment is straight in that system, and any consumer reprojecting it inherits a line that was never on the ground.

Stored as a densified geodesic, the file is larger by the factor the tolerance table gives and is correct in every plane a consumer might use. That is the only one of the three that survives reprojection, and it is the reason interchange formats carry so many vertices for lines that are conceptually two points.

None of this arises for a short line in a well-chosen local system, which is designing a grid for one region: over a survey’s extent the departure is millimetres and the question does not come up. It arises exactly when a dataset is global, which is exactly when it is delivered on a web map.

The navigational form of the same problem came first by three centuries, and it behaves identically. A route flown as a small number of constant-heading legs departs from the great circle it approximates by an amount quadratic in the leg length, so a handful of legs buys most of the accuracy — the same economics the vertex table above reports, arrived at by people who had no files and no reprojection.

The rhumb line, which is a straight segment in one plane and a real route #

There is one plane in which a straight segment is not an artefact but a navigational instruction, and it is the reason the projection under all of this exists.

A straight line on Mercator is a rhumb line: a course of constant compass bearing. It is not the shortest route — the departure measured above is exactly how much longer and how far off it runs — but it is the route a vessel actually steers, because holding a bearing is something a helm can do and following a geodesic requires continuous alteration. Why Mercator exists is the essay on that property, and it earns the projection its place.

So the transatlantic segment in the hero figure is not meaningless. It is a rhumb line from New York to London, and a ship steering it would arrive, 718 kilometres of cross-track and some extra distance later. What makes it a defect is that nothing in the file says rhumb line — the segment was stored as two points by somebody meaning “the way between these”, and the plane supplied a specific and unrequested interpretation.

That is the cleanest statement of this essay’s claim. The straight segment is not wrong; it is answering a different question, and the question it answers is the one its coordinate system asks.

What the picture is claiming #

The final point is the site’s own premise, applied to a line rather than to a map.

A drawn segment is a statement that the ground between the endpoints is there. It has the same standing as any other mark on a figure, and on this site every mark is required to be something computed rather than something assumed. A straight line between two stored points is the one mark in common use that is drawn without anybody deciding what it means — supplied by the renderer, inherited from the plane, and read as a route.

The shortest route is not straight makes the argument about the world; this makes it about the file. The two endpoints are exact, the arithmetic is exact, and 718 kilometres of the Atlantic is a claim nobody made.

One more consequence of the third-order law is worth stating for the densification table. Cutting a line into n pieces divides the cross-track error by n² and the extra distance by n³, so a densification chosen to meet a positional tolerance overshoots the length tolerance by a wide margin — thirty-two pieces buy a five-hundred-metre departure and a length error of centimetres. A specification written in metres of position is always the binding one, and nobody needs to write a second one about length.

Where this ladder goes #

The next rung is the one place in this field where the coordinate system’s failure is not a matter of degree: the line where longitude stops counting up, at which a bounding box becomes the whole world and a midpoint lands at the antipode of where it belongs.