A tolerance in map units is not a tolerance

Eleven rungs of this ladder have priced the geometric operations a dataset gets put through — an area, a centroid, an intersection, a buffer, a nearest-neighbour query. Each of them is a computation on projected geometry, and each essay’s answer has the same shape: the computation is performed on the page and the reader wants the ground.

This one is not about a computation. It is about a parameter.

Every one of those operations takes a number: a snapping distance, a cluster radius, a simplification tolerance, a minimum feature size, a buffer width. The number is in the units of whatever coordinate reference system the geometry happens to be in, and a projected unit is a different ground distance at every point.

The conversion is the scale factor #

There is no subtlety in the arithmetic. A tolerance of $t$ map units at a point buys $t/a$ metres of ground in the direction the projection stretches most, and $t/b$ in the direction it stretches least, where $a$ and $b$ are the two principal scale factors — the semi-axes of Tissot’s indicatrix, which this collection has been computing since its first essay.

No radius of the Earth appears anywhere in that, which is the sign that the quantity really is a ratio. What makes it worth an essay is not the formula but where the formula gets applied, which is nowhere.

The rule gets stricter toward the poles, which is the opposite of what almost everybody assumes. The intuition is that Web Mercator inflates high latitudes, so surely everything is bigger up there — and it is, on the page. A fixed page tolerance therefore buys less ground where the page is inflated, so a global snapping pass cleans Africa generously and Norway barely at all.

The exception that proves the mechanism #

Mercator’s ground tolerance is exactly $t\cos\varphi$ times its equatorial value, and it is checked against that closed form to a part in a million at every latitude in the ladder.

Web Mercator’s is not. It departs from the cosine law by up to 0.39 per cent, and the reason is this site’s founding measurement: Web Mercator applies spherical formulae to ellipsoidal latitudes, so its scale is not exactly $\sec\varphi$ against the ellipsoid, and it is not conformal.

That is a 0.39 per cent effect inside a 470 per cent one, so nobody would notice it here. It is worth stating because it is the same defect turning up in a completely different quantity, and because it means the tolerance is not even circular — which is the next section.

The tolerance is an ellipse #

A snapping operation compares distances, and a distance on the page is a circle’s worth of tolerance in every direction. On the ground that circle is the preimage of a circle under the projection, which is an ellipse with axis ratio $a/b$ — the projection’s own anisotropy.

So on a plate carrée dataset a five-unit snapping tolerance merges features up to 6.37 metres apart east–west and only 3.19 metres apart north–south. The operation is directional and the direction is the map’s, which no documentation of any snapping tool mentions, because from inside the tool the tolerance is a scalar.

The conformal members return exactly one to five decimal places, which is the control: conformality is precisely the statement that the tolerance is a circle.

And it depends on which file it is #

The factor between coordinate systems at one latitude is 2.5, and it stacks with the factor of 5.7 across latitudes within one. A tolerance chosen for a dataset in one system and applied to the same features stored in another is doing a materially different job, and there is nothing in either file that says so.

This is the same defect the operation decides the coordinate system is about, arriving through the parameter rather than through the geometry. That essay’s rule is that an operation should choose the system it is performed in; this rung adds that the operation’s parameter has to be converted when it does, and a pipeline that reprojects the data and keeps the number has silently changed the rule.

The repairs, in order of cost #

Four, and the order is the reverse of how much attention each gets.

Convert the parameter when the data is reprojected. One multiplication by the local scale factor, and it removes the failure exactly at the point where it is introduced. Nothing in any pipeline tool does it automatically, because a parameter is not tracked as a quantity with units.

State the tolerance in ground metres and convert at use. The tolerance becomes a property of the job rather than of the file, so a dataset moved between systems keeps the same rule. It costs a scale-factor evaluation per feature and it is exact.

Snap on the ellipsoid. The operation itself moves off the page, which removes every projection effect at once, including the anisotropy. It is what computing an area needs a surface recommends for areas, and it is available in every serious library and default in none.

Or split the work by region. A dataset processed country by country in each country’s own grid has a tolerance that means the same thing throughout, because a national grid is conformal to parts per million over its own area. That is what everybody did before global datasets, and it is why this failure is recent.

What the accident looks like #

The failure is not that a tolerance is wrong. Every value in the table above is a correct conversion of a correctly stated number. The failure is that the topology a pipeline produces depends on latitude, and topology is the one output nobody checks numerically.

Two features merge, or they do not. A snapped dataset has a different number of vertices, a different number of nodes, and a different set of shared boundaries depending on where in the world each feature is. A boundary that two features share is about what happens when two neighbours disagree about their common edge; this is about a pipeline that decides they have a common edge in Africa and decides they do not in Norway.

And the decision is irreversible. Snapping discards the original coordinates. A dataset cleaned with a five-unit tolerance cannot be un-cleaned, and the record of what was merged is not kept, so the latitude dependence is not recoverable afterwards even in principle.

The threshold latitude is not round. 52.84° north, for the stated pair and the stated tolerance, and it is exactly $\arccos(3/4.967)$ — a number produced by two decisions made in different places by different people, neither of whom was choosing a latitude. That is the shape of the whole failure: a boundary in the data that nothing intends, nothing records and nothing documents.

The boundary is one of many, and it moves fast #

The threshold at 52.84° is derived for one separation and one tolerance, and generalising it says something worse than the single number does.

A pair separated by d metres merges wherever the tolerance’s reach exceeds d, so the boundary sits at

$$\varphi = \arccos\!\left(\frac{d}{t\,r_0}\right),$$

with r₀ the ground metres one map unit buys at the equator — 0.9934 for Web Mercator. Every distinct separation present in the data therefore has its own parallel, and a global dataset contains a continuum of them: pairs a metre apart merge up to 78°, pairs three metres apart up to 52.8°, pairs four metres apart up to 36.4°, and pairs beyond 4.97 metres never merge at all.

So a snapped global dataset is not divided into a clean north and a clean south. It is stratified, with a different subset of its near-coincident pairs surviving in each latitude band, and the strata are set by the distribution of separations in the source data rather than by anything anybody chose.

The boundaries also move faster than the parameter does. Differentiating gives dφ/dt = cot φ / t, which at the worked point is 8.7 degrees of latitude per map unit — and a direct evaluation confirms it: raising the tolerance from five units to six moves the three-metre boundary from 52.8° to 59.8°, seven degrees, or seven hundred and eighty kilometres of ground.

That is the sensitivity a pipeline owner is actually exposed to. Changing a tolerance from five to six is the kind of adjustment made without a note, and it relocates the line at which the dataset’s topology changes by the width of a large country. Nothing downstream can detect the move, because both datasets are internally consistent and neither records where its own boundary is.

Everything comes from distortion, which is the same function every other essay here uses, evaluated at stated points. The tolerance conversion is a division. No dataset, no coastline and no simulation appears anywhere.

Three assertions carry the rung and each rejects a different way of being wrong.

The ground tolerance must fall with latitude on Web Mercator, by more than a factor of five across the range a global dataset covers, and the spherical Mercator’s must follow $\cos\varphi$ to a part in a million. A sign error in the conversion — dividing where it should multiply — passes the first check and fails the second, which is why both are there.

The tolerance must be exactly circular on a conformal projection and an ellipse of ratio at least two somewhere in the library. The first is what conformality means and a failure would mean the indicatrix is being read wrongly; the second is the finding.

And one tolerance must merge a stated pair at some latitudes and not at others. A version where every latitude behaved the same would mean the essay has nothing to report.

Where the model stops #

A tolerance in a geographic coordinate system is worse, not better. Degrees are not a unit of length at all, which is a degree is not a unit of length’s whole subject, and a tolerance of 0.0001° buys 11.1 metres east–west at the equator and 1.9 at eighty degrees — the same failure with a larger constant and a much more obvious cause.

This is a point property. The tolerance’s reach varies across a feature as well as between features, so a long line crossing thirty degrees of latitude is snapped by a rule that changes along its own length. Nothing here measures that; it would need the variation integrated along a path, and the effect is bounded by the endpoint values.

Simplification is a different case with the same arithmetic. A tolerance is a promise about the picture argues that a simplification tolerance is legitimately a page quantity, because what it promises is that the drawn line will not visibly move. That is right, and it is exactly why the two must be told apart: a page promise should be in page units and a ground promise should not, and both are stated as one number in the same field of the same tool.

Nothing here is about the accuracy of the data. A coordinate is a number with a width prices what a positional error does to a snapping decision, and that is a separate and much better understood effect — a tolerance is usually chosen to be the positional accuracy, which makes the two the same size and makes this rung’s variation a variation in whether the tolerance still matches the data it was chosen for.

And nothing here concerns projected coordinate systems designed for a region. A state plane zone or a national grid is conformal over its own area with a scale factor within a few hundred parts per million of one, so the effect is parts per million and the answer is that for the job the system was designed for, a tolerance in its units is a tolerance. The failure belongs to global datasets in a global projection, which is what a web map’s coordinate reference system is.

One number that is safe #

There is a tolerance in this subject that does not have the defect, and naming it sharpens what the defect is.

An angular tolerance — a maximum bend, a minimum interior angle, a collinearity test — is dimensionless, so it has no units to be in. It is still not preserved, because only a conformal projection preserves angles, but its failure is bounded by the projection’s own angular deformation and is therefore a property of the map rather than of the latitude. On a conformal system it is exactly right.

That is the distinction to carry: a tolerance in the units of a coordinate system inherits the scale factor and varies by a factor of eleven; a tolerance in angle inherits only the angular deformation and is exact on a conformal system. Where a parameter can be stated as an angle it should be.

The generalisation #

The rule is short: a parameter is a quantity and it lives in a coordinate system too.

This collection’s standing rule is that an operation performed after the map is an operation performed on the map, and every rung of this ladder has applied it to data — a coordinate, a length, an area, a centroid, a query result. A parameter is not data. It arrives from a configuration file, or a function argument, or a form field, and nothing about it looks like geometry.

But a tolerance is a length, a cluster radius is a length, a minimum mapping unit is an area, and a maximum slope is an angle — and the slope of a field that was measured prices what a projection does to that one. Every one of them is a measurement in the units of a coordinate system, and every one of them is written down once and applied everywhere. That is a worse exposure than the data has, because data is at least reprojected when it moves and a parameter is copied.

The practical rule follows: when a pipeline changes coordinate system, its parameters need converting as surely as its geometry does, and a parameter that cannot be converted — because it was chosen against a picture rather than against the ground — needs saying so.

Who found it, and when #

Nobody, in the sense that everyone doing this work knows that projected units vary. It is stated in every introductory text and it is the reason the advice “use a projected coordinate system for distance operations” exists at all.

What that advice does not say is which projected coordinate system, or what to do when the data is global. Web Mercator is the default storage system for a very large fraction of the world’s vector data, precisely because it is what web maps display, and it is the worst possible system for a distance parameter — its scale factor varies by a factor of eleven over the latitudes it covers.

The literature that would have caught this is the literature on positional accuracy standards, which is careful about ground distances and has nothing to say about tool parameters. And the literature on generalisation is careful about tolerances and treats them as page quantities, correctly, for its own purposes. The failure lives in the gap: a page tolerance used as a ground promise, by a tool whose documentation calls it a distance.

Where the ladder goes next #

Twelve rungs have taken the geometry and now the parameters. What has not been touched is the third thing a pipeline carries, which is the order the operations are performed in: snapping before or after simplification, reprojection before or after a buffer, a dissolve before or after a clip. Each of those pairs has a preferred order, most of them do not commute, and the amount by which they fail to commute is a measurement nobody makes.