The renderer runs out of numbers before the zoom does

Every quantity this ladder has computed — a pyramid of tiles, a zoom that is a ladder, a tile drawn without its neighbours — a tile extent, a pixel’s ground size, a quantisation step, the width of a drawn road — has been computed in double precision and reported as though the renderer had it.

It does not. A graphics pipeline carries vertex coordinates in single precision: twenty-four bits of significand, about seven decimal digits. A Web Mercator world coordinate near the antimeridian is a number of order $2 \times 10^7$ metres, and the spacing between representable single-precision values there is two metres.

Why this one is different from every other error here #

Everything else in this ladder gets better as the zoom increases. The pixel shrinks; the tile covers less ground; the quantisation step of a stored coordinate falls; the ground error of a two-pixel road width falls with it.

This one does not move. The world coordinate stays the same size at every zoom, so the gap between representable values stays the same, and the pixel it has to be drawn into shrinks by half each level. In pixel units the error therefore doubles with every zoom level.

0.05 pixels at zoom 12. 0.84 at zoom 16. 3.3 at zoom 18. 13.4 at zoom 20. 53.6 at zoom 22.

The crossover, and where it sits #

The crossover — the zoom at which the representable step first exceeds one pixel — is at zoom 17 near the antimeridian. Web map clients routinely offer zoom 20 to 22.

Past the crossover a vertex snaps to a lattice coarser than the screen. A straight road acquires a staircase; two lines that should meet do not; a polygon boundary jitters as the view pans, because the same ground position rounds to different lattice points depending on what else is on screen.

And it depends on where the coordinate is #

This is the part that is genuinely surprising, and it follows immediately from how a floating-point number works: the spacing between representable values is proportional to the magnitude, so a coordinate of 20 million is represented a thousand times more coarsely than a coordinate of 20 thousand.

Nothing about the ground is different at the two places. The map is the same map, the projection is the same, the data has the same accuracy. What differs is the size of the number standing for the position, and the size of the number is decided by where somebody put the origin of the coordinate system.

So a web map is more precise near longitude zero, by seven zoom levels, and the reason is a choice made in the 1880s about where to put a prime meridian.

The fix, and why the format looks the way it does #

Every vector tile format stores geometry on an integer grid inside the tile. The convention is 4,096 steps across a tile, and that integer grid has consequences of its own which this ladder has already priced.

What that essay treats as a storage decision is also a precision one, and it is the more important half. The largest number the renderer ever handles is 4,096 rather than 20,037,508, and single precision represents every integer up to 16,777,216 exactly — so a tile-local coordinate has no representation error whatever, at any zoom, anywhere in the world.

The tile’s own origin is added in double precision on the way in, or is folded into the transformation matrix, and never meets the vertex data at all.

What a float actually is, in one paragraph #

The behaviour above follows from one property of the representation and it is worth stating rather than assuming.

A binary floating-point number is a significand times a power of two. Single precision has 24 bits of significand, so within any interval $[2^k, 2^{k+1})$ it can represent exactly $2^{24}$ values, evenly spaced at $2^{k-23}$. The spacing is therefore proportional to the magnitude, doubling at every power of two.

At a magnitude of $2 \times 10^7$, which is between $2^{24}$ and $2^{25}$, the spacing is $2^{25-24} = 2$ metres exactly. That is where the two metres comes from, and it is why the number is a round one: it is a power of two, not a rounded measurement.

Double precision has 52 bits, so at the same magnitude the spacing is $2^{25-52}$, which is $7.5$ nanometres. The two formats are separated by a factor of 268 million, which is the whole difference between a world that fits and one that does not.

Two errors that look alike and are not #

There is a quantisation in the format and a quantisation in the arithmetic, and they are easy to conflate.

The format’s quantum is the tile grid: 4,096 steps across a tile, which at zoom 16 is 15 centimetres of ground and at zoom 20 is 9 millimetres. It shrinks with the zoom, because a tile covers less ground at higher zooms, and it is the subject of the earlier rung.

The arithmetic’s quantum is the floating-point spacing, and it depends only on the magnitude of the number being held. Held tile-locally it is zero. Held as a world coordinate it is two metres and does not shrink.

So the format’s choice does not merely reduce the arithmetic error; it eliminates it. That is a much stronger property than a smaller number, and it is why the two are worth separating.

What goes wrong when the fix is not applied #

The failure appears in exactly the places where the tile-local discipline is broken, and there are several.

A vertex transformed on the client. A geometry converted to world coordinates before being handed to the pipeline has already been rounded to the world-coordinate lattice, and no later transform recovers it.

An overlay drawn from raw coordinates. A route, a marker path or a user-drawn shape supplied in projected metres and passed to the renderer directly is a world coordinate, and it jitters against a tiled basemap that does not.

A computed geometry, where reprojecting invents values on top of everything else. A buffer, an intersection or an interpolation performed in single precision on world coordinates loses to the same lattice, and the loss shows as slivers at the boundary between two shapes that should be coincident.

The jitter, which is the visible symptom #

A static screenshot rarely shows the problem. Motion does, and the way it shows is worth describing because it is how the defect is usually first noticed.

Panning a map changes which part of the world is on screen and, in some pipelines, changes the translation applied before the vertex reaches the pipeline. A vertex whose world coordinate rounds to one lattice point under one translation can round to a neighbouring one under the next, so the drawn position moves by a quantum between frames.

At half a pixel that is invisible. At fifty-four pixels it is a line that walks. The characteristic appearance is a geometry that is stable while nothing moves and shifts in discrete jumps when the view does, which reads as a bug in the panning code and is not.

The same mechanism produces the other familiar symptom: two coincident boundaries, drawn from two layers, that agree in one view and separate in another. Both round to the lattice, and whether they round the same way depends on the translation each layer happened to be handed.

Why nobody sees it below zoom 17 #

The error has been present in every renderer since the first one and is rarely reported, and the reason is that most of the time it is smaller than a pixel.

Below the crossover the lattice is finer than the screen, so a vertex snapping to it moves by less than a pixel and nothing is visible. Above it the movement is visible, and above zoom 20 it is unmistakeable — which is precisely the range that only became routine when maps started being used for indoor plans, cadastral overlays and construction drawings.

So the bug is as old as the technique and became a defect when the use case arrived. That is a common shape and it is worth naming: an approximation is not wrong until somebody zooms in.

The other half of the number #

Everything above is about the spacing between representable values. There is a second quantity — how many values there are — and it points the same way.

Single precision has 24 bits of significand, so it distinguishes about 16.8 million values within a binade. A Web Mercator world spans about 40 million metres, so representing it to the metre needs 25 bits and representing it to the centimetre needs 32.

That arithmetic is the whole argument in one line: the world does not fit in a float. It fits in a double comfortably — 52 bits of significand is nanometres on a planet — and the pipeline that draws it is not made of doubles.

What a coordinate’s width means here #

A coordinate is a number with a width makes the general argument: a stored coordinate has a precision, the precision is a length on the ground, and quoting more digits than the width supports is a claim nobody can support.

This rung is the case where the width is imposed by the machine rather than by the survey or the format. It is not a decision anybody made about how much precision the data deserved; it is what is left after the number has been put into a register, and it is invisible in the file, in the format specification and in every part of the pipeline before the last one.

A precision that depends on a nineteenth-century decision #

The origin dependence deserves one more paragraph, because it is the most quotable consequence and it is easy to state carelessly.

The claim is not that maps of Britain are more accurate than maps of New Zealand. The data is not more accurate; the ground is not better surveyed; the projection behaves identically. What differs is the number of bits left over after the magnitude of the coordinate has been accounted for, and that is decided by the distance from the coordinate system’s origin.

Web Mercator’s origin is the intersection of the equator and the prime meridian, and the prime meridian is where it is because of the 1884 International Meridian Conference. So the seven-zoom-level difference in representable precision between longitude 1° and longitude 179° is, in a literal and slightly absurd sense, a consequence of a diplomatic conference.

The absurdity is the point. A choice of origin is supposed to be a convention with no consequences — a grid has an origin that is not there argues exactly that for survey grids — and here it has one, because the arithmetic that stores the coordinate is not translation-invariant.

What this collection’s own figures do #

It is worth noting where this site stands, since it draws a great many coordinates.

Every figure here is an SVG, and SVG coordinates are decimal text parsed into whatever the browser uses, which is double precision. The page coordinates a generator emits are of order a few hundred, so the representable spacing is around $10^{-14}$ and nothing here is close to the limit.

That is not a virtue; it is a consequence of drawing at page scale rather than at world scale. The moment a figure emitted world coordinates and let a renderer transform them, it would inherit everything above — which is one reason every figure here does its own projecting rather than handing world coordinates to something else, and names the projection it is drawn in while it is at it.

Where the model stops #

The spacing figures here are computed from the floating-point format directly and are exact. What they do not model is what a particular pipeline does with them: some renderers carry a per-tile translation in the vertex shader, some carry a double-precision matrix and split it into two single-precision parts, and some do the transformation on the processor and hand over screen coordinates.

Each of those changes the answer and none of them changes the constraint. Whatever the arrangement, the number that reaches the rasteriser is single precision, and the question is only how small the pipeline managed to make it before it got there.

What to check, if this is suspected #

The symptom is distinctive enough to diagnose without instrumentation, and three checks separate it from the alternatives.

Does it depend on the zoom rather than on the data? A geometry that is clean at zoom 16 and stepped at zoom 21, with no change to the source, is an arithmetic problem rather than a data one. A coordinate stored coarsely is coarse at every zoom.

Does it depend on the longitude? The same geometry, translated to a different part of the world, should behave differently if the cause is the magnitude of the world coordinate. Seven zoom levels between the meridian and the antimeridian is a large and easily observed difference.

Does it move when the view moves? A lattice that the geometry snaps to, rather than a fixed error in the geometry itself, produces motion under panning. A wrongly-stored coordinate does not move; a wrongly-computed one does.

Those three between them separate this defect from a coarse source, a bad reprojection and a rounding in the tile format, which are the three things it is usually mistaken for.

Who found it, and when #

The technique of rendering in a local frame is older than web maps and comes from astronomy and flight simulation, where the same problem arrives with much larger numbers. The standard treatment is to translate the world so the camera is at the origin, which makes the numbers near the viewer small and the numbers far away irrelevant.

Web mapping arrived at the same answer by a different route: a tile is a natural local frame, and storing geometry relative to it was adopted for compression rather than for precision. That it also solves the precision problem completely is a coincidence of the kind that suggests the format was designed by people who had met the problem before.

The coincidence at the end of that account is worth one more sentence, because it has a hazard in it. A guarantee that arrives as a side effect of a decision made for another reason is a guarantee nobody wrote down — so nothing prevents a future revision of the format, made for compression or for streaming or for anything else, from moving the coordinates back into a global frame and silently reintroducing the defect. The property is real, it is load-bearing, and it is currently protected only by the fact that the format has not changed. Documenting it costs a sentence and converts luck into a requirement.

It is worth saying which sentence. Not coordinates are stored relative to the tile, which is already documented as a compression measure, but the tile-local frame is what keeps single-precision rendering exact at high zoom, and any change to the frame must preserve that. The first is a description of the format and the second is a requirement on it, and only the second survives somebody optimising the format for a different goal.

The same reading applies to this collection’s other found guarantees: a property that holds because of how something happens to be built is worth writing down as a requirement precisely when nobody has had to think about it.

Where the ladder goes next #

Nine rungs have priced the pyramid, the pixel, the tile and now the arithmetic. What remains unasked is the thing between the tile and the screen: the transformation itself, which on a tilted or rotated view is not a scaling and a translation, and which has its own precision behaviour.