The pixel is a place with a size

A coordinate in a file is a point: two numbers with as many digits as the format carries. A coordinate on a screen is a pixel, which is a square of ground with a size. Getting from the first to the second is a rounding, and like every rounding on this site it has a magnitude that can be computed and a consequence that is not the obvious one.

The bound #

A renderer works in whole pixels, so a vertex is moved to the nearest pixel centre. The move is at most half a pixel in each direction, so at most half a diagonal in total:

$$d_{\max} = \frac{\sqrt2}{2}\, r(z)\cos\varphi$$

At level 12 that is 16.8 metres, at level 16 it is 1.05, and at level 19 — the deepest a general-purpose scheme usually goes — it is 13 centimetres. The site’s gate checks the bound directly rather than deriving it: it quantises real coordinates at four latitudes and requires every displacement to fall under the half-diagonal, which fails if the pixel grid has been mis-scaled anywhere in the arithmetic.

The direction of the latitude effect, which is the wrong way round #

The pixel covers less ground at high latitude, because the ground resolution is $r(z)\cos\varphi$, so the quantisation is finer where the map is more stretched. At 60° a level-12 pixel is 19.1 metres of ground against the equator’s 38.2, and the displacement bound halves with it.

That is worth stating explicitly because the intuition points the other way. A Mercator map at 60° north looks coarse — everything is inflated, a country occupies four times the paper it should — and the natural inference is that positions there are held loosely. The opposite is true: the inflation is extra pixels, and extra pixels are extra precision. A Norwegian city at level 12 is drawn to twice the positional resolution of an Ecuadorian one.

The site’s assertion writes that down in the form that can fail: the ratio of the ground resolutions at two latitudes must be exactly $\cos\varphi_1/\cos\varphi_2$, checked to $10^{-9}$, so a version of the calculation that had dropped the cosine would pass the bound check above and fail this one.

Whether two things are two things #

Here is where the quantisation stops being a rounding error and becomes a property of the scheme.

Two points 43 metres apart, at level 11 where a pixel is 76 metres: is that one dot or two? The obvious answer — one, since they are closer together than a pixel — is wrong, and the first version of this site’s assertion failed on it. The pair straddled a pixel boundary and came out as two dots.

Whether they merge depends on the phase between the pair and the grid, which is a property of where the tiling’s origin happens to fall and has nothing to do with the data. So the honest quantity is a rate over phases, and it has a closed form: the pixel indices agree in $x$ for a fraction $1 - |\Delta x|/p$ of offsets and in $y$ for $1 - |\Delta y|/p$, independently, so the merge rate is the product.

For this pair at level 11 the scan over 1,600 offsets gives 12 per cent and the closed form gives 12.4. One level deeper the pixel is 38 metres, smaller than the separation, and the rate is exactly zero for every offset — which is the only guarantee available.

A separation smaller than a pixel does not mean a merge; a separation larger than a pixel does mean a separation. The asymmetry is the finding, and it is the reason the assertion demands all three: zero above the threshold, the predicted rate below it, and a rate strictly between zero and one, which is what a version that had lost the phase scan could not produce.

Two grids, and the one that is not there #

A stored coordinate is quantised too, and on a different grid. Six decimal places of a degree is a uniform step in degrees; a pixel is a uniform step in projected metres. The obvious question is which is the coarser, and the obvious answer is that it depends on the latitude for both.

It does not, and the measurement is the sharpest thing in this essay.

East–west the two grids stay in constant ratio. A degree of longitude shrinks as $\cos\varphi$ and a Mercator pixel covers ground as $\cos\varphi$, and it is the same $\cos\varphi$: the projection’s scale factor is exactly the reciprocal of the convergence of the meridians it compensates for. Measured across nine latitudes from the equator to 80°, the ratio varies by 0.333 per cent and no more.

And what is left of it is not noise. That residual is $N(\varphi)/a$, the ellipsoid’s own transverse radius of curvature against the sphere the tiling uses — measured 0.33271 per cent, predicted 0.33271 per cent, agreeing to nine decimal places. It is the site’s headline defect arriving for the third time in this field, and asserting the agreement rather than the smallness is what turns a leftover into a measurement.

North–south they cross. A degree of latitude is a fixed ground distance — 110.6 to 111.7 kilometres over the whole ellipsoid — while the pixel keeps shrinking as $\cos\varphi$. So there is a latitude at every zoom below which the file is the coarser grid and above which the screen is. At level 16 with five decimal places stored, it is at 62.19°, found by bisection on the difference rather than from a spherical closed form.

The order of the two roundings matters #

Two quantisations on grids that are not aligned do not commute, and the size of the disagreement is a rate rather than a distance.

Storing a coordinate to five places and then drawing it, against drawing it and then snapping to a pixel, puts the vertex in a different pixel for 13.3 per cent of positions along a parallel at 52.5° north at level 16. The prediction is 11.7 per cent, and the factor in it is worth keeping: snapping to the stored grid displaces a point by a sawtooth of amplitude half a step, whose mean magnitude is a quarter of a step, and the pixel changes when that displacement crosses a boundary. Predicting the ratio of the step sizes instead — the first version — is four times too high and looks like a real disagreement between theory and measurement.

Store to four places instead and the rate is 100 per cent: every position moves. That is the refusal the assertion carries, because a comparison that had quietly become a comparison of something with itself would report zero at every precision and pass a check that asked only for a small number.

What follows for a file #

Three things, and none of them is use more decimal places.

A pixel coordinate is not a coordinate. It has a zoom level baked into it and is only meaningful with that level attached, which is why nothing stores one. What is stored is degrees, and the pixel is computed at the moment of drawing — the operation this whole field is about.

The right number of decimal places is a function of the deepest zoom served, not of the data’s accuracy. At level 19, half a pixel is 13 centimetres at the equator, so five decimal places — 1.11 metres — is the limiting quantisation, and the picture is coarser than the file by a factor of eight. Six places puts the file below the screen everywhere. Seven is storing noise, in the precise sense that no rendering can distinguish it.

And the two grids are different kinds of object. The stored grid is uniform in the coordinate; the pixel grid is uniform on the projected plane, which is uniform on the ground only where the scale factor is one. That distinction is the same one the units are part of the coordinate makes about survey feet and metres, arriving through a completely different door.

What the picture does to a survey coordinate #

The practice field spends four essays getting a coordinate to a centimetre. It is worth asking what survives being drawn.

What a tape measures reduces a 77-kilometre line through four corrections and finds a 583-millimetre term that a job’s own tolerance makes significant. The ground is not the grid computes the combined factor to parts per million. A published coordinate is a result explains why a control point’s coordinate is held fixed by convention rather than re-observed.

Drawn at level 16, all of that is quantised to a metre. At level 19 — deeper than any general scheme serves — it is quantised to 13 centimetres. A centimetre survives only at level 23, which is four levels past the bottom of the pyramid and would need 70 trillion tiles to tabulate.

So a screen map cannot be evidence of a position at survey precision, at any zoom anybody serves, and the reason is not accuracy but representation: the picture has cells and the coordinate does not. That is the same boundary the tolerance decides the model draws for corrections — a term matters when it exceeds the tolerance of the job — read in the opposite direction. Here the display sets a floor, and it is a floor no amount of care in the survey can get below.

Why nobody notices, which is the interesting part #

Every vertex on a screen map has moved by up to half a pixel diagonal, and nothing looks wrong. The reason is worth stating because it is what makes the quantisation acceptable rather than merely tolerated.

The rounding is deterministic and shared. Two polygons meeting along a common boundary have their shared vertices rounded by the same function to the same pixel, so the boundary stays shared and no gap opens. A quantisation that was even slightly path-dependent — that rounded a vertex differently depending on which feature it arrived in — would open a hairline crack along every administrative boundary in the world, and the effect would be instantly visible.

And the displacement is below the ink. A stroke is one to four pixels wide, so a half-pixel move is inside the line that draws it. The picture is not claiming to place anything more precisely than its own strokes, and the quantisation respects that bound exactly.

What is visible, occasionally, is the failure of the first property across a different grid: two datasets prepared in two coordinate systems and overlaid, where the rounding happens after two different transformations and the shared boundary is no longer shared. That is two grids over the same ground in the form a reader meets it, and the crack it opens is a real disagreement between the two systems rather than an artefact of drawing.

Why this is not a rounding error #

It is tempting to file all of the above under floating point, which would be a mistake of the same kind as filing the scale bar under printing.

A rounding error is an unavoidable consequence of finite arithmetic and is bounded by the machine’s precision. What is happening here is a change of representation: a point becomes a cell, and cells have a size chosen by a scheme for reasons of delivery. The bound is 16.8 metres at level 12 not because a double cannot hold more, but because somebody decided a tile is 256 pixels and a level is a power of two.

The signature that distinguishes them is the phase dependence. Floating-point error does not depend on where an arbitrary grid’s origin falls; this does, completely — the same two points are one dot or two according to a choice made in a specification. That is the strongest form of the field’s recurring claim: the answer is a measurement of the operation as much as of the ground, and here the operation includes a constant nobody thinks of as part of the data at all.

The same question at the other end of the site #

A cell with a size, a quantity that is only defined as an average over it, and a claim that gets stronger as the cell shrinks: the pattern is not new here.

How small is flat enough inverts a quadratic bound to find the patch size at which a plane survey meets a stated tolerance. A screen map is a pyramid of tiles has the pyramid’s levels as the same kind of ladder in the other direction. And the resampling essays later in this field measure a kernel’s error by refining a grid and fitting the exponent, which is the same question — how fast does the cell’s size stop mattering — asked of an image instead of a survey.

What is peculiar to the pixel is that its size is set by a delivery convention rather than by the ground or the instrument. A survey patch’s size comes from curvature and a tolerance; a raster cell’s from the sampling; a pixel’s from the decision that a tile is 256 across. The first two are answerable to something. The third is answerable to a specification, which is why the merge rate depends on a grid origin and why the whole effect can be moved by half a pixel by a choice nobody records.

A delivery convention that became a geometric parameter #

The tile being 256 pixels across is the quiet centre of this rung, and it is worth separating what kind of number that is.

It was a delivery decision. A tile had to be small enough to fetch quickly over a slow connection and large enough that a screen did not need too many of them, and 256 was a reasonable answer to that in the early 2000s. Nothing about the Earth, the projection or the data was consulted.

It is now a geometric parameter of the world’s most-used map. The pixel’s ground size at every zoom level, the merge rate this rung measures, the precision at which a coordinate survives a round trip through a rendered tile — all of them are functions of that number, and every one of them is a statement about position rather than about delivery.

And it has become effectively unchangeable. Tile sizes of 512 exist and are used, and switching is not a matter of a setting: zoom levels stop lining up between providers, cached tiles are the wrong size, styling breaks, and every stored zoom-to-scale table is wrong by a factor of two. A convention adopted for bandwidth is now load-bearing for geometry, which is the ordinary fate of a number that ends up in an interface.

The half-pixel is the same story one level down. Where the grid’s origin sits relative to the cell centres decides which pairs of places merge, it is a choice made independently by every producer, and no raster format has a field that records it unambiguously. The effect can be moved entirely by half a cell, and the half-cell is not written down.

Which gives the modest proposal this rung supports. Not that anybody change the tile size — that ship has sailed and the argument for changing it is weak — but that the origin convention be recorded, since it costs a field and it decides an effect this essay measures.

That is the smallest change that would make the measurement in this rung reproducible by somebody holding the data rather than the pipeline.

Where this ladder goes next #

The pixel is where a coordinate stops being a point. The next rung is where a feature stops being one thing: a tile is rendered from the geometry inside it and knows nothing of its neighbours, so a polygon spanning nine tiles gets nine labels and none of them where its centre is.