Rounding is not noise

Eight rungs of this anchor give a coordinate a width and then treat that width as noise: independent between points, near enough Gaussian, shrinking as the root of the number of observations. Every one of those assumptions is right about a measurement, and every one is wrong about the other error a published coordinate carries.

A coordinate is written to a stated number of digits. The difference between what is written and where the place is has nothing random about it — the same place always rounds the same way — and it is shared by every point that falls in the same cell.

On one coordinate they are the same thing #

The reason the substitution is so easy to make is that on a single point it is correct.

Rounding to $d$ decimals gives an error uniform on half a cell either side, with standard deviation $\text{step}/\sqrt{12}$. At five decimals and 45° that is 0.3210 metres, and the measurement over two hundred thousand positions returns 0.3209. A Gaussian of that σ makes the same prediction for everything that involves one point at a time: the expected offset, the probability of being within a stated distance, the contribution to a variance.

So an error budget that lists “coordinate precision, 0.32 m” alongside “observation noise, 0.32 m” and adds them in quadrature is doing the right arithmetic. The two part company the moment a second point is involved.

It is worth being precise about the one part of the coincidence that is a coincidence. The $\sqrt{12}$ is not a convention: the rounding error is exactly uniform on the cell, so its variance is exactly $\text{step}^2/12$, and a σ quoted as half a cell — which is what “accurate to half a metre” usually means when somebody counts digits — is 1.73 times too large. The measurement above returns 0.3209 against a predicted 0.3210, which is the third digit and is the check that the distribution is uniform rather than merely bounded. A quantiser whose error was not uniform, because the data clustered on cell boundaries, would fail that check and would behave differently everywhere below.

Below a cell they are opposite #

At two hundred metres apart the two models give 0.392 and 0.395 metres of distance error and are interchangeable. At five centimetres they give 0.239 and 0.514, and the numbers understate how different they are, because the shape of the error is different: the rounded distance is very often exactly zero and the noisy one never is.

Ninety-three per cent of pairs five centimetres apart are the same point in a file written to five decimals. Nothing in a noise model produces that, and nothing about the size of the error hints at it: the standard deviation is the same 0.32 metres in both cases.

Which one is right about a short distance #

Neither, and the directions matter more than the magnitudes.

A pair five centimetres apart has a true distance of five centimetres. Rounding returns zero for ninety-three per cent of such pairs, which is a hundred per cent relative error and a five-centimetre absolute one. Noise returns something scattered around half a metre, which is a thousand per cent relative error and a half-metre absolute one.

So under rounding a short distance is too short and bounded, and under noise it is too long and unbounded, and the two failures call for opposite defences. Against rounding the defence is to refuse to report a distance shorter than a cell, because the answer carries no information — a zero means the same cell, not the same place. Against noise the defence is to report the distance with its uncertainty, which will exceed the distance itself and say so.

The difference of two coordinates prices the noise version of this and finds the difference’s error is √2 times a single coordinate’s. That factor is the noise model’s answer and it is right for noise. Under rounding the factor is zero for points in one cell and √2 for points several cells apart, with a transition in between — so the anchor’s own formula for a difference is the large-separation limit of something that has two regimes.

Averaging separates them completely #

At $n = 1$ the two are 0.3203 and 0.3191 — indistinguishable. At $n = 1{,}024$ they are 0.3155 and 0.0100, a factor of thirty-two apart, and the factor is exactly $\sqrt{n}$ because that is what the noise model buys and the rounding does not.

The error that does not average down makes the neighbouring argument about time: observations of one place are correlated in time, so the mean falls more slowly than the root. This is the same failure in space and it is more extreme, because the correlation is not partial. Two readings of one place have a rounding error that is not merely correlated but identical, so the mean of any number of them has exactly the error one of them has.

The one thing rounding does better #

The comparison is not all one way, and the exception is worth having because it is what makes rounding usable at all.

Rounding is bounded, which is the property the tolerance that decides the verdict is really about wherever a threshold has to be defended. A coordinate written to five decimals is within 0.556 metres of its true place, always, with no exceptions and no tail. A Gaussian of the same standard deviation is outside that bound 8.2 per cent of the time and has no bound at all.

That matters for the operation an error budget is usually built for. A statement that a point is inside a tolerance is a proof if the only error is rounding and a probability if the error is noise, and the two need different language in a specification. A coordinate is a number with a width argues that a position is an interval rather than a point; the interval is genuinely an interval when the error is quantisation and is only a confidence region when it is not.

What the cell actually is #

Everything above is at 45°, and the cell is not square anywhere except the equator. That is the anchor’s own first rung and it compounds with this one: a pair of places separated east–west collapses at a different separation from a pair separated north–south, and the ratio between the two is $\cos\varphi$.

So the collapse threshold is direction-dependent as well as latitude-dependent. Near the pole two places can be a metre apart east–west and share a written longitude while two places a metre apart north–south do not share a written latitude.

The consequence for a dataset is that its resolution is anisotropic in a way nobody declares. A file of coordinates at five decimals holds a metre of north–south resolution everywhere and between a metre and ten centimetres of east–west resolution depending on latitude, so the same file distinguishes places better in one direction than the other and better in the tropics than in Scandinavia. A user who describes it as “accurate to about a metre” is describing one number out of three.

The case this is really about #

The abstract comparison is worth grounding in the operation that actually goes wrong, because it is a common one and it has a name.

Deduplicating a point dataset means finding records that refer to the same place, and the usual test is a distance threshold. Under a noise model the threshold is chosen from the noise: a few standard deviations, so that genuine duplicates fall inside and distinct places fall outside, with a stated error rate either way.

Under rounding the test behaves differently at both ends. Genuine duplicates written from the same source are bit-identical, so they are found by an exact match with no threshold at all. Distinct places closer together than a cell are also bit-identical, and no threshold can separate them — the information is gone from the file. And places between one and a few cells apart have a distance error that is not small relative to the distance, so the threshold’s error rate is not what the noise calculation says.

The number of distinct places a file can represent is the number of cells it covers, and no processing recovers more. At five decimals that is one point per square metre or so, which sounds generous until the data is about building corners or survey marks — where a metre is the wrong unit entirely and the cell has already merged things the survey distinguished.

What was computed, and how #

Every comparison runs the two models side by side on the same pairs, with the noise given exactly the rounding’s own standard deviation of $\text{step}/\sqrt{12}$. That equality is not incidental; it is what makes the comparison about the structure of the two errors rather than about their size, and it is checked before anything else.

The pairs are placed at a random position and a random bearing at a stated true separation, and the sphere is treated as locally flat, which at these separations is right to far better than the effect being measured.

The generator had to be replaced. A hash of the loop index was the first version, and its low bits correlate across consecutive indices — so the two uniforms feeding a Box–Muller transform were not independent and the resulting “standard normal” had a standard deviation of 3.9. The noise arm of every comparison came out four times too wide, which reads as rounding being four times better than noise everywhere and is a statement about the generator rather than about coordinates. It is a stateful mulberry32 now, and the noise arm’s own standard deviation is checked against its nominal value.

The assertions require six things separately: that the rounding error of one coordinate have the predicted standard deviation; that a substantial share of pairs well inside a cell collapse to a single point under rounding and none under noise; that the two agree beyond twenty cells; that averaging leave the rounding error unmoved at every count while reducing the noise as the root; that the rounding error never exceed half a cell while the noise exceeds it measurably; and that the two disagree by more than a factor below a fifth of a cell.

What this collection’s own numbers do #

The audit is short and it is the reassuring direction for once.

Nothing in these essays is computed from published coordinates. Every position here is either a stated place given to four decimals as an input — London at −0.1278, 51.5074 — or a point generated by the machinery at full double precision. So no figure on this site has a quantisation error in it at all, and the cell measured above is a property of the files the essays are about rather than of the essays.

The one place it does bite is the input side. Those stated places are written to four decimals, which is a cell of 11.1 metres north to south — and every distance this collection computes between two of them inherits it. The pooled score abandons a region works in region centres and is unaffected at any precision that matters; the datum sensitivities are rates per kilometre of height and are unaffected; the reach sets are hundreds of kilometres. Eleven metres is below the last digit printed in every one of them, which is a fact worth having checked rather than assumed.

Where the model stops #

Rounding of a projected coordinate is a different lattice. Writing an easting and a northing to the metre carves the ground into squares in grid units, whose ground image is the indicatrix acting on a square — a parallelogram whose area is the metre’s own divided by the areal factor. quantisationCell computes both and the anchor’s first rung compares them; everything here is about the geographic case, which is the one a published latitude and longitude is.

Real data is rounded and noisy. The two errors are independent and add in quadrature for anything involving one point, so the total is $\sqrt{\sigma_{\text{obs}}^2 + \text{step}^2/12}$ — and every structural statement here survives, because the noisy part averages down and leaves the rounded part behind. The regime that matters is the one where the quantisation dominates, which is exactly where a coordinate has been published to fewer digits than it was surveyed to.

And the collapse is a property of a file, not of a place. Two points that share a written coordinate are distinguishable by anything else in the record — a name, an identifier, an attribute. What is lost is only the geometry, and it is lost irrecoverably: no processing of the file can separate two points it holds as one.

The generalisation #

The rule is that two errors of the same size are not the same error, and the thing that distinguishes them is what happens when they are combined.

An error budget is a list of standard deviations, and standard deviations add in quadrature regardless of where they came from. That arithmetic is right for a single quantity and it throws away the structure — whether the errors are independent between points, whether they are bounded, whether they survive averaging — and every one of those is a question about an operation rather than about a number.

The same shape runs through this collection wherever an estimator has been audited. A maximum over a sample has the same standard deviation as a noisy measurement and a sign the noise does not have. The page-sampled mean has a bias where the ground-sampled one has none, at the same spread. In each case the number that gets quoted is a magnitude and the thing that bites is a structure.

The habit is one question, and it costs nothing: before adding two errors, ask what each would do to a difference, to an average, and to a bound. If they answer differently, they are different errors that happen to be the same size, and the budget that adds them has already lost the information a reader needs.

Who found it, and when #

Quantisation error is thoroughly understood in signal processing, where it has been analysed since the 1940s and where the distinction between it and additive noise is standard — the uniform distribution, the $\Delta^2/12$ variance, the correlation between successive samples of a slowly varying signal, and the dither that is added precisely to break that correlation and make quantisation behave like noise.

None of that vocabulary has crossed into geodesy, and the reason is that the two disciplines meet the problem at opposite ends. A signal processor chooses the quantisation and can design around it; a surveyor receives a file someone else wrote and has no control over how many digits it has. So the literature that has the theory has no cause to talk about coordinates, and the literature about coordinates treats precision as a number of digits rather than as an error model.

The one place the profession does behave correctly is in specifying how many digits to publish, where the guidance is explicit and is expressed as a ground distance rather than as a count. That guidance is about the size of the cell. What is missing is what the cell does to a difference, an average and a bound — which is everything a coordinate is used for after it has been read.

The cheap habit: if a coordinate came out of a file, its error is bounded and its neighbours share it. Do not average it, do not trust a distance shorter than a cell, and do treat the tolerance check as exact — which is the one thing the situation gives back.

Where the ladder goes next #

Nine rungs have priced what a coordinate’s width does: to a position, to an average, to a difference, to a length, to an area and now to a pair of places that are too close together to tell apart. Every one of them has taken the digits as given. The number of digits is itself a decision, and the format a coordinate is stored in makes it — which is a second quantiser, on top of the one somebody chose.