The scale factor of a line

A grid’s scale factor is a number attached to a point. A survey measures the distance between two points, and there is no reason for the scale factor at either of them to be the number that converts one distance into the other.

What the conversion actually needs is the average of the scale factor along the line, weighted by arc length. That is an integral, and every rule in the survey manuals for approximating it is a statement about the shape of the integrand.

The point value is not useless — it is what the ground is not the grid combines with the elevation factor, and over a short line it is the whole answer. What follows is about where “short” stops.

Why it is a parabola #

In a transverse Mercator the scale factor depends on the distance from the central meridian and, to a very good approximation, on nothing else:

$$k \;\approx\; k_0\left(1 + \frac{E^2}{2R^2}\right)$$

where $E$ is the easting measured from the central meridian and $R$ is the radius of curvature. The quadratic is the whole of the behaviour at survey scale; the next term is fourth order in $E$ and contributes parts per billion across a zone.

Two consequences follow immediately and they are the essay.

Along an east–west line the integrand is a quadratic in the parameter. The easting varies linearly with position along the line, and $k$ varies with the square of the easting, so $k$ traces a parabola from one end to the other.

Along a north–south line the integrand is very nearly a constant. The easting barely changes, so neither does $k$. What variation remains comes from $R$, which changes by half a per cent between the equator and the pole and by far less over any one country.

The three rules #

The endpoint mean, $(k_A + k_B)/2$, is what a spreadsheet does and what a great deal of software does. It replaces the curve by the chord joining its ends.

The midpoint value, $k_M$, is what a hurried job does: compute the scale factor once, at the middle, and use it.

Simpson’s rule, $(k_A + 4k_M + k_B)/6$, is what every survey manual gives, and it is exact for any cubic — which means exact for the parabola the transverse Mercator produces.

The manuals rarely say why the third is preferred, and the reason is not that it uses more points. It is that the integrand’s shape is known and Simpson’s rule matches it exactly. A rule chosen to match the integrand is a different kind of thing from a rule chosen to be more careful.

What the three rules actually cost #

The comparison has to be in metres on the line, because parts per million are not a quantity anybody’s specification is written in and because the interesting question is whether a rule is adequate rather than whether it is best.

The endpoint mean is out by 18 millimetres on a 556-kilometre north–south line, which is inside any tolerance a line that long would be held to. On a 362-kilometre east–west line — a shorter line, on the same grid, at the same latitude — it is out by 97 metres.

Ninety-seven metres. Not a rounding, not a subtlety, and not a consequence of the line being long, because the line it happens on is the shorter of the two.

The expectation that was wrong #

This essay was planned around the claim that a long line needs the better rule, and the measurement refused it.

The intuition is reasonable and it is the one every rule of thumb in the field encodes: specifications say things like apply Simpson’s rule to lines over ten kilometres, which is a statement about length. The measurement says length is the wrong variable. A north–south line of any length in a transverse Mercator zone can be handled by the endpoint mean, and an east–west line a fifth as long cannot.

What decides is the direction, because the direction decides whether the line traverses the parabola or sits at one point on it. A specification keyed to length will pass jobs it should fail and fail jobs it should pass, in a pattern determined by the local road network.

This is the second time in this collection that a rule of thumb has turned out to be keyed to the wrong variable. The first was UTM’s scale factor, which reduces the worst-case error across a zone at the equator and increases it above 57.5° north, because the rule is stated in degrees of longitude and the quantity it controls is a distance.

The size of the effect, against the grid it lives on #

Ninety-seven metres sounds implausible until it is put beside the quantity being averaged.

Across a zone the scale factor runs from about 400 parts per million below unity on the central meridian to a thousand above it at the edge — a spread of fourteen hundred parts per million, which on a 362-kilometre line is five hundred metres of difference between the two ends’ worth of correction. Getting the average of that wrong by a fifth is not a subtle failure; it is a plain consequence of replacing a parabola by its chord, and the chord of a parabola sits at two-thirds of the height rather than at the average.

The three rules land at predictable places on that parabola. The midpoint value is too small by a third of the sag, the endpoint mean is too large by two-thirds of it, and Simpson’s rule weights them 4:1:1 and lands on the answer. Those are the classical facts about a parabola and they explain every number in the comparison.

What a specification should say instead #

If length is the wrong variable, the right one is available and is no harder to state: the change in easting along the line.

That is the quantity the parabola is a function of, so it is the quantity that decides how much of the curve a line traverses. A rule keyed to it would say something like use Simpson’s rule whenever the line’s easting changes by more than twenty kilometres, and would be right in every direction, at every latitude, on every transverse Mercator grid.

No specification the collection has seen is written that way. They are written in terms of line length, which correlates with easting change only for a line of unknown bearing — which is to say, on average over all the jobs a specification will ever be applied to, and never for the job in front of anybody.

What was computed, and how #

The integrand, sampled at 61 points along the line by evaluating the transverse Mercator’s own derivatives at each — not by interpolating a formula for $k$, because the point of the exercise is to measure what the projection does rather than what the approximation says it does.

The reference, by composite Simpson at 256 panels. Every rule’s error is measured against that.

The convergence rate, which is the assertion that matters. Simpson’s rule is fourth order, so halving the panel width should divide the error by sixteen. The measured ratios are 16.01 and 16.01.

That rate is asserted rather than the error’s size, and the reason is one this collection keeps rediscovering: a check that only demands a small error passes just as happily when the integrand has quietly become a constant. A constant integrates exactly at every panel count, so it produces no rate at all — and a check on the rate fails in exactly the case a check on the size cannot see.

The repair the first version of that check needed #

The convergence check was first written to compare each estimate against the 4,096-panel reference, and it reported ratios of 27.0 and then 1.3.

The 1.3 is not a failure of Simpson’s rule. It is the third estimate arriving at the reference’s own noise floor, so the quantity being divided had stopped being truncation error and become the difference between two nearly identical floating-point sums. A rate measured against a reference is only a rate while the estimates are far enough from it to have one.

The repair is to compare each estimate against the next refinement rather than against a fixed reference — a self-difference, which has no floor until the estimates themselves stop moving. That is the standard way to read a rate off a quadrature and it is why the numbers now come back at 16.01 rather than at 27 and 1.3.

Worth recording because the first version passed on its first two ratios. A check that measures the right thing badly can be green for a long time.

Where the difference goes #

A hundred metres of error in a line scale factor does not appear as a hundred-metre coordinate error. It appears as a hundred metres of disagreement between a distance computed from coordinates and a distance measured on the ground, which is a different and more insidious thing.

Coordinates derived through the wrong rule are internally consistent. Every distance between them is wrong by nearly the same proportion, so the figure they describe is the right shape and the wrong size — the signature that a traverse must close shows a closed traverse is completely blind to. The error surfaces when the work meets somebody else’s, or the ground, and not before.

Where the line scale factor sits in the chain #

The line scale factor is one of four multiplications between a tape reading and a coordinate, and it is not the largest of them.

Being the second-largest term does not make the averaging rule less important. It makes it more important, because the largest term — the slope reduction — is a trigonometric identity with no approximation in it at all, so it cannot be got wrong by a choice of method. The line scale factor is the largest term in the chain that admits a wrong answer.

Where the model stops #

The line is sampled in grid coordinates, which is a choice. Parameterising along the geodesic instead answers a slightly different question — the average of $k$ weighted by distance on the ellipsoid rather than on the map — and the two differ in the sixth significant figure on the lines here. That is far below the differences between the three rules and it is stated rather than hidden, because it is exactly the kind of unstated choice that makes published comparisons of this sort unreproducible.

The quadratic is an approximation. The exact scale factor of a transverse Mercator comes from the Krüger series and carries terms in $E^4$ and beyond; what is drawn here is the projection’s real derivatives, so the parabola is a description of the picture rather than the source of it. On a zone six degrees wide the fourth-order term reaches a few parts per billion, which is why the description holds.

Nothing here applies to a conic. A Lambert conformal conic’s scale factor varies with latitude and is constant along a parallel, so every claim in this essay reverses: a north–south line needs Simpson’s rule and an east–west line does not. The structure survives, the direction does not, and a practitioner moving between the two families with the habit rather than the reasoning will be wrong in exactly the way this essay is about.

The generalisation #

An average is not a value, and a rule for computing one encodes an assumption about shape. That is true of every quadrature, and what makes the cartographic case worth drawing is that the shape is known in closed form — so the correct rule is not a matter of prudence but of matching, and the wrong rule’s error can be predicted rather than bounded.

It also puts a familiar piece of advice in its place. Use more points is the universal remedy for a numerical disagreement, and here it is nearly useless: the endpoint mean does not converge to the right answer by being applied to more points, because it is not a rule about points but a rule about the whole line. Simpson’s rule at three points beats the endpoint mean at any number.

The rule a designer can remove instead #

There is a way to make the choice of averaging rule stop mattering, and a few grids take it.

If the zone is narrow enough, the parabola is short enough that all three rules agree. That is the argument behind the narrow high-accuracy zones the United States uses and behind designing a grid for one region generally: shrink the extent and every downstream question gets easier, at the cost of needing more zones and meeting more of the seams between them.

The observation worth keeping is that these two design pressures are different. A grid is usually narrowed to hold the scale factor’s magnitude down; narrowing it until the scale factor’s variation along a line stops mattering is a stricter requirement, and no published grid the collection has looked at states which of the two it was designed against.

Who found it, and when #

Thomas Simpson published the rule in 1743, and it was already old — Kepler used it, and Bonaventura Cavalieri had it a century earlier. Its arrival in survey practice is much later and thoroughly practical: computing a scale factor by hand at three points was affordable, and computing it at thirty was not, so the rule that gets the right answer from three points was the one that mattered.

That constraint is gone. A modern computation could evaluate the integrand a thousand times and never notice, which makes it worth asking why the three-point rule persists in specifications. The answer is that it is exact, not approximate, for the integrand a transverse Mercator produces — so refining it buys nothing, and the specification is right for a reason its authors could state and its readers usually cannot.

A specification that is right for a reason nobody states #

The persistence of the three-point rule is unusual in this collection, because most surviving conventions are outlived approximations and this one is not.

Simpson’s rule is exact here rather than approximate. The integrand a transverse Mercator produces is a quadratic in the distance along the line, and Simpson’s rule integrates a quadratic exactly — so three evaluations give the same answer as three thousand, to the last digit the arithmetic carries.

Which means the specification cannot be improved by refining it, and the ordinary complaint about an old rule of thumb does not apply. A surveyor following it is not accepting an error for convenience; they are using the cheapest exact method available.

The awkward part is that almost nobody knows this. The rule is taught as a formula to apply, in the same register as the approximations around it, and a practitioner asked whether evaluating at more points would be more accurate would reasonably say yes. So the specification is right, its readers cannot say why, and the thing they cannot say is the difference between a rule that should be kept and one that should have been retired with the slide rule.

And the distinction is not academic. A modernisation exercise reviewing an old specification has to decide which of its rules are exact and which are approximations that survived their constraint. The exact ones should be kept as they are; the approximate ones can be improved for nothing. Telling them apart requires knowing what each rule’s integrand is, which is exactly the knowledge that gets lost when a method becomes a procedure.

This collection’s version of the same point is that a claim is worth a test it could fail. Simpson’s rule is exact for this integrand is such a claim: refine the quadrature, and if the answer moves, the claim is false. It does not move, and that is the whole of the justification the specification never states.

Where this goes next #

The scale factor along a line is what a survey needs. The scale factor a grid has is a number its designers chose, and the choice can be reconstructed: the scale factor was chosen, by an optimisation with a closed form. Where that reasoning is run again for a region of one’s own, it produces a grid designed for one region.