The rotation has two sign conventions

This ladder has taken a seven-parameter datum transformation apart eight times. It has priced what each parameter does, where a fit leaves residuals, what happens when two are chained, what another common point buys and how uncertain the numbers themselves are. Every one of those treats the seven numbers as given and asks what they do.

None of them asks what the numbers mean, and three of the seven mean two different things.

Two matrices, one transformation #

A Helmert transformation applies a translation, a scale and a small rotation — and a datum is fitted to a region before it has any of them to a geocentric Cartesian position. The rotation is small enough — under three arcseconds for every datum in ordinary use — that it is written to first order as a matrix that differs from the identity only in its off-diagonal entries.

There are two ways to write that matrix, and both are in print:

$$R_{\text{PV}} = \begin{pmatrix} 1 & -r_z & r_y \\ r_z & 1 & -r_x \\ -r_y & r_x & 1 \end{pmatrix}, \qquad R_{\text{CF}} = R_{\text{PV}}^{\mathsf T}.$$

The first is the position vector convention: the point is rotated. The second is the coordinate frame convention: the axes are rotated, which is the same physical operation performed the other way round. They are transposes, and for a rotation a transpose is an inverse, so the same physical transformation is published with rotations of opposite sign depending on which convention its publisher used.

The two have EPSG method codes — 9606 and 9607 — and that is the only place many parameter sets record which they are. A parameter set copied out of a report, a wiki, a forum answer or a software configuration file is seven numbers and a name, and the convention is carried in prose or nowhere.

What the mistake costs, exactly #

Applying one convention’s formula to the other’s numbers is not a small error and it does not partly cancel. The difference between the two answers at a point $X$ is

$$R_{\text{PV}}X - R_{\text{CF}}X = 2\,s\,(r \times X),$$

to the precision of the linearisation. Three things follow from that expression and each of them is a fact about the shape of the error.

It is twice the rotation, not the rotation — so the intuition that a sign error costs “about as much as the parameter is worth” is wrong by a factor of two.

It is perpendicular to the rotation axis, so it is a turn of the whole body rather than a shift.

It is proportional to the distance from that axis, so it vanishes on the axis and is largest ninety degrees away.

The numbers #

OSGB36’s published rotations are 0.1502″, 0.2470″ and 0.8421″, whose magnitude is 0.8903″ — about $4.32 \times 10^{-6}$ radians. Twice that times the Earth’s radius is 55 metres, and the worst point of the world is out by exactly that.

DHDN’s are 0.202″, 0.045″ and −2.455″, a magnitude of 2.4637″, and the worst point is out by 152 metres.

The other two datums in this collection’s table publish translations and nothing else, so their two conventions agree exactly and the confusion is impossible. That is not a general reassurance: a parameter set with no rotations is a parameter set whose fit was constrained to have none, and the modern realisations of most national datums have them.

For a working point in Britain the OSGB36 confusion is 28 metres, against a datum shift whose own size is about 99 metres. So the error made by reading the parameters wrong is more than a quarter of the correction they exist to apply — large enough to matter to anything, and small enough to look like a plausible answer.

Why it is not caught by a sanity check #

A 28-metre error in Britain is large, and the reason it survives is that nothing about it looks wrong.

It is not a factor-of-ten mistake, which a glance catches. It is not a hemisphere flip, which a map catches. It is not a units confusion, which a magnitude check catches. It is a coordinate that lands 28 metres from where it should, in the right field, in the right town, on the right side of the river — and 28 metres is inside the error a user of a datum transformation is often willing to attribute to the transformation itself.

That is the trap’s real shape. The published accuracy of a national seven-parameter transformation is usually quoted at a few metres, sometimes worse; a user who expects a few metres and gets twenty-eight has a discrepancy they can explain to themselves. The size of the confusion sits in the band where a wrong answer is indistinguishable from a poorly performing correct one.

The pattern is what makes it findable #

Every practical defence against this rests on the shape of the error rather than on its size.

A single common point catches it, provided the point is not near the axis: transform a known coordinate both ways and compare. That is a two-minute check and it is the reason the mistake is usually caught in a well-run pipeline.

What it does not catch is a pipeline with no common point at all, which is the normal state of a small job. Somebody takes a coordinate off a map, finds a transformation on the internet, applies it, and has no independent value to compare against. The answer is wrong by 28 metres and looks entirely reasonable.

The residual on the axis is the flattening #

The check above has a small surprise in it worth keeping.

On the rotation axis the cross product vanishes, so the two conventions should agree exactly. Measured, they agree to 0.113 metres — small against 55 and not zero. The reason is that the axis is a direction in geocentric Cartesian space, and the point of the ellipsoid at the longitude and latitude the axis points to does not lie on it.

That is because a geodetic latitude is measured along a normal that misses the centre by up to 21 kilometres, which is the whole content of geodetic against geocentric latitude arriving four essays away from where it was proved. The 0.113 metres is the flattening, showing up in a place nobody would look for it, and its ratio to the off-axis value is 487 to one.

The blind cone #

The blind spot exists and is not a practical worry, which is worth saying because a reader who has followed the argument this far will expect it to be one.

At a two-centimetre tolerance the cone within which the confusion is invisible has an angular radius of a hundredth of a degree for OSGB36 and less for DHDN. A check point has to be within about a kilometre of a place that has nothing to do with the survey — for OSGB36, 59° east and 71° north, which is in the Kara Sea — before it fails to notice.

So the practical rule is the simple one: one common point catches it, anywhere on Earth that anybody works. The failure mode is having no common point at all, not having a badly chosen one.

Why this is not the same as the direction problem #

There is an older and better-known trap next door and the two get conflated.

The inverse of a Helmert transformation is not its negation: the scale and the translation do not commute, so undoing the seven parameters in the wrong order costs about a centimetre for OSGB36. That is a direction error — applying the transformation the wrong way round — and its size is second order in the parameters.

The convention error is first order and is four thousand times larger. They are frequently discussed together because both are described as sign problems, and they are not the same problem: one is about which way the transformation runs, the other about what its middle three numbers mean.

What the metadata should carry, and usually does not #

The transformation is seven numbers, a source datum, a target datum, and a method. The method is the part that is dropped.

EPSG’s registry records it — code 9606 for position vector, 9607 for coordinate frame — and a parameter set taken from the registry through a library that understands the codes is safe. A parameter set typed into a configuration file, quoted in a report’s appendix, or pasted from a forum is a list of numbers, and there is no way to recover the convention from the numbers themselves.

That is the honest summary. The convention is not deducible from the parameter set; it is metadata, and it is the piece most often lost. Two parameter sets can describe one transformation makes the argument that the numbers are not unique; this one makes the sharper argument that they are not even self-describing.

A detection that works without a common point #

There is one weak signal available when nothing else is.

Published parameter sets for a given datum pair are usually derivable from one another. If two sources give translations that agree and rotations that are equal and opposite, they are the same transformation under the two conventions, and the disagreement is resolved. If two sources give rotations that agree exactly, they share a convention and one of them may have copied the other’s mistake.

That is a consistency check rather than a determination, and it fails silently in the case that matters most — a single source, quoted once, with no sibling to compare against.

What a rotation of one arcsecond is worth #

The arithmetic is worth doing once in plain terms, because the parameters are quoted in arcseconds and arcseconds are not intuitive.

One arcsecond is $4.848 \times 10^{-6}$ radians. Multiplied by the Earth’s radius that is 30.9 metres of displacement at the point furthest from the axis. So a rotation parameter of one arcsecond moves a coordinate by about thirty metres, and getting its sign wrong moves it by about sixty.

Set against the translations, which are hundreds of metres, the rotations look like a refinement. They are not: OSGB36’s three rotations together contribute 27 metres of the correction and DHDN’s contribute 76, and in both cases that is the difference between a transformation that meets its published accuracy and one that does not.

That is the reason the seven-parameter form exists at all. A three-parameter shift is enough where the rotations are negligible and is not enough anywhere the fit needed them, and a user who has been given seven numbers has been given them because three were insufficient.

Where the model stops #

Everything here uses the linearised rotation, which is what every published parameter set is expressed in. For rotations of a few arcseconds the neglected second-order term is a fraction of a millimetre on the Earth’s radius, so the closed form $2s(r \times X)$ is exact to well past the precision anybody works to.

What is not treated is the case where the two conventions are combined with a direction error as well, which is four combinations rather than two and can produce an answer that is nearly right by cancellation. That is a real failure mode in badly assembled pipelines and it needs a different figure — a two-by-two of the four possibilities — which this rung does not draw.

The check this site’s own gate makes #

The assertion shipped with this rung is in two halves and both are necessary.

The first requires that the measured difference between the two conventions match the closed form $2s(r \times X)$ at every sampled point, to a millimetre. That is a comparison between a number obtained by running two transformations and differencing the results, and a number obtained by evaluating a cross product — two routes that share no arithmetic, which is the only reason to believe either.

The second is the refusal, and it is the half that would catch a broken implementation. A parameter set with no rotations must produce exactly zero difference between the conventions, at every point of the world. Both of the translation-only datums in the table return zero to $10^{-9}$ metres, so the machinery is capable of reporting that the confusion is impossible and does so whenever it is.

A version of the check that omitted the second half would pass on a bug that returned a large number everywhere, which is precisely the kind of bug a cross-product implementation is prone to.

Who found it, and when #

The two conventions are both older than the confusion. The position-vector form comes from the geodetic literature and is the one the IERS uses; the coordinate-frame form comes from the practice of rotating reference frames rather than points and is common in national and military documentation. Both were in use before anybody exchanged parameter sets electronically, and neither is wrong.

The confusion is an artefact of the exchange. When transformations were applied by the organisation that fitted them, the convention travelled with the people; when they became seven numbers in a file, it did not. EPSG’s separation into two method codes is the fix and it dates from the registry’s own maturity in the 1990s, which is to say that the problem had to exist first.

One sentence of metadata #

The remedy is not technical and it is not new. A transformation should be quoted as nine things rather than seven: the source datum, the target datum, the seven parameters, and the method code that says which convention the middle three are in.

Every registry that matters does this. Almost every secondary source that quotes a transformation does not, because the method code looks like bookkeeping and the seven numbers look like the content. A parameter set without it is not a transformation; it is two transformations, one of which is 55 metres wrong, with no way to tell them apart.

Why the convention did not travel #

The account of how the confusion arose contains a general rule about metadata, and it is worth extracting because it predicts where the same failure will occur next.

A convention travels when it lives in the same artefact as the data it governs. While transformations were computed and applied inside one organisation, the convention lived in the practice — in the code, the training and the people — and never had to be written down, because it was never separated from its numbers.

Publication separates them. Seven numbers can be copied into a paper, a textbook, a forum answer, a configuration file or a spreadsheet; the convention cannot, because it is not one of the numbers. So every act of copying strips it, and the copies look complete because seven parameters is what a seven-parameter transformation has.

Which is the rule: a datum that is not a field will be lost at the first copy. The method code is a field in the registry and is therefore carried by anything reading the registry properly, and is dropped by every secondary source that quotes the parameters as a list. That is not carelessness; the list looks like the whole thing.

And the same shape is visible elsewhere in this collection. A coordinate without its reference system, a rate without the geometry it was divided by, a derived raster without the plane it was computed in, a fitted aspect without the symmetry convention that disambiguates it — in every case a quantity that is unambiguous in the place it was produced becomes ambiguous the moment it is written down, because the disambiguating information was in the room rather than in the file.

The remedy is always the same and always sounds trivial: make it a field. The reason it keeps not happening is that at the moment of publication the omission is invisible to the person best placed to fix it, since they are the one person for whom the convention is obvious.

Where the ladder goes next #

Nine rungs of this anchor have treated a transformation’s parameters: what they do, how well they are known, how they compose and now what they mean. The unasked question is what happens when there is no transformation at all — when two datums have no published relation and the only way from one to the other is through a third.