Which of these numbers are the sampler's

Four rungs of an anchor built to be sceptical about this collection’s own instruments have each found something. That leaves a question the anchor cannot avoid and would be dishonest to leave unanswered: how much of what these essays have published is actually a statement about the sampler?

The measurement is straightforward. Take the quantities this collection prints, recompute them at three times the sample density, and report what moves. Nothing is being tested against a closed form here — most of these have none — so nothing that follows is a proof of correctness. It is a sensitivity measurement, and the point of it is the split, which is much cleaner than the four rungs below would predict.

The split #

Two columns, two behaviours. The means barely move and every one of them moves in whichever direction the arithmetic happens to take it. The maxima move a great deal and every single one moves up, which is the first rung’s sign, arriving on this collection’s own numbers.

That is the whole of the audit’s bad news, and it is enough to be worth stating plainly. Every worst-case distortion this collection has printed is a lower bound, some of them by nearly a fifth. A reader told that the Miller cylindrical’s worst angular deformation over the sphere is 50.5° has been given a number that becomes 62.4° when the sampler is refined by a factor of three, and would climb further still.

The refinement column and the shortfall figure agree about the direction and disagree about the size, and the disagreement is instructive: refining from twenty samples to sixty recovers most of the shortfall but not all of it, so the numbers in the right-hand column of that table are lower bounds too. Where the worst point is ranks projections by this quantity, and the whole of that ranking sits on the wrong side of the truth by an amount that varies from map to map.

The good news, and why it is not luck #

The essays do not argue with individual numbers. They argue with comparisons: this projection is better than that one for this purpose, this criterion and that criterion disagree, this rule of thumb fails on this region. The object those arguments are made of is a ranking.

The ranking is stable. From twelve samples a side — a coarse setting, well below anything used here — the order of eight projections over the whole sphere is byte-identical at every density tried, with all twenty-eight pairs in the same relation.

The reason is worth stating carefully, because it is the thing that makes the previous four rungs survivable. The ranking is built from means, and means converge fast. The Kavrayskiy number is a quadrature of a smooth bounded integrand, so its error falls as the square of the spacing, and by twenty samples a side it is in the third decimal place. The gaps between projections are in the second. A statistic whose error is an order of magnitude below the differences it is being used to detect will get the order right, and stay right.

The counterfactual #

That is a claim about the choice of statistic rather than about good fortune, and the way to establish it is to make the other choice and look.

Ranking by the maximum rather than the mean puts an estimator with a large, size-dependent, one-directional bias in charge of the order, and the order moves: two inversions at eight samples a side, one at twelve, none above. That is a much milder effect than the size of the individual movements would suggest, and the reason is the one thing the first rung did not measure.

The maxima all move up together. Their shortfalls are large but they are correlated: refining the sampler adds between 2 and 19 per cent to every projection’s worst point, and the ordering survives because most of the movement is common to all of them. What breaks an ordering is not a large bias but a large spread in the bias, and the spread here is smaller than the gaps it would have to cross for six of the eight pairs it could affect.

So the honest summary is a conditional. A ranking by a mean is safe at any density this collection has used. A ranking by a maximum is safe above about twenty samples a side and is not safe below it, and the published literature contains comparisons made on much coarser grids than that.

What was computed, and how #

The audit runs regionDistortion twice for each projection over the whole sphere, at twenty samples a side and at sixty — 1,600 points against 14,400, a factor of nine in cost — and reports the relative change in four quantities. Nothing else changes: the same projections, the same region, the same code, the same derivatives.

Two projections in the table are not in the list above, and both are instructive. Mercator’s Kavrayskiy number moves by 0.02 per cent and its mean angular deformation appears to move by 8.6, which is arithmetic noise: it is conformal, its angular deformation is zero everywhere to a part in a million, and a relative change in a quantity that is zero is not a measurement. Its worst areal error moves by 48 per cent, for the reason the second rung gives — the quantity is unbounded, so refining the sampler simply moves the outermost sample closer to a pole, and there is no limit for the sequence to approach.

The assertion carrying the rung has four parts. No summary mean may move by more than two per cent under the refinement; the maxima must move at least four times as much as the means, so that the contrast is real; the mean-based ranking must be stable at every density; and the maximum-based one must not be, because a rung whose counterfactual behaved identically to its case would have established nothing.

What has been changed #

Three things, and the first is the only one that touches an existing page.

Every worst-case number in this collection is now reported with its refinement. A sampled maximum is stated as a sampled maximum, with the value a search that can reach the boundary returns beside it, wherever the two differ by more than the precision printed. That is the first rung’s repair and it costs a pattern search per figure.

A ranking states the density it was computed at. Not because the rankings here move — they do not — but because a reader has no way to tell a ranking that is stable from one that has never been checked, and the two look identical.

And nothing has been changed about the means, which is the finding rather than an omission. Six of the eight Kavrayskiy numbers here move in the fourth decimal place under a ninefold increase in sampling, and refining them further buys nothing anybody could use.

The step under all of it #

Every number in this anchor, and every number in the two hundred and ninety essays around it, is computed from a Jacobian, and every Jacobian here is a finite difference with a step somebody chose. That is a sampling decision of exactly the same kind, made one level below all the others, and an audit that stopped short of it would have stopped one level short.

The left arm has fitted slope 4.01, which is a surprise only to someone who has not read jacobian. A plain centred difference is second order; this one takes two centred differences at $h$ and $h/2$ and combines them by Richardson’s construction, which cancels the leading term and leaves the fourth power. The comment beside it says why, and the reason is a good one: conformality is tested by asking whether two principal scale factors are equal, so the noise floor of the derivative sets the smallest non-conformality this collection can detect, and a plain difference left Mercator reading 1.2 × 10⁻⁶ degrees of angular deformation, which is close enough to a real tolerance to be uncomfortable.

So the one sampling decision at the bottom of the whole stack was audited on the day it was made, against a stated requirement, and the ladder above confirms it: the optimum sits at 3 × 10⁻⁴ and the default of 10⁻⁴ is within a factor of three of it, at 2.6 × 10⁻¹² of relative error. Fourteen orders of magnitude below anything else in this anchor.

That is worth ending on, because it is the shape of what the audit found overall. The estimator this collection thought about is fine. The estimators it never thought about — the maximum, the mean, the point set, the refinement — are the ones with something wrong with them, and every one of them was inherited rather than chosen. The tolerance that decides the verdict makes the same observation about a threshold: the numbers somebody argued over are sound, and the ones that arrived as defaults are the ones to check.

Where the model stops #

This is a sensitivity, not an error. Comparing an estimate with a finer estimate of the same kind says how much it is still moving, which is a lower bound on how wrong it is and not a measurement of it. The four rungs below all made a point of comparing against something outside the estimator; this one cannot, because there is nothing outside for most of these quantities, and the whole rung is therefore weaker than the ones it follows. That is stated rather than hidden.

Eight projections and one region. The audit is over the whole sphere, which is the most demanding case for a mean and the easiest for a ranking, because the differences between projections are largest there. Over a small region the gaps narrow, so the same sampler error is a larger fraction of them, and a ranking over a small region is correspondingly more fragile. That is measured for regional comparisons and is not measured here.

Nothing here checks the figures. Every generator in this collection computes at its own density for its own picture, and those densities were chosen for drawing rather than for accuracy. A figure that reads a number off a coarse sample and prints it in its caption is the same failure in a place the audit does not reach, and finding it needs a sweep of every caption rather than a sweep of every projection.

And the fourth rung’s failure is invisible here by construction. A comb that a doubling ladder never touches is a comb this audit never touches either, since it doubles and then triples. Robinson is in the table and its worst angular deformation moves by 15 per cent, which is the ordinary boundary effect; the table entries contribute nothing to it, because they contribute nothing to any maximum over latitudes the grid visits and nothing at all to a mean.

The generalisation #

The rule that comes out of this anchor is not measure more finely. Three of the five rungs have a case where refining does not help and one has a case where it makes matters worse.

The rule is: know which side of the answer the estimator sits on, and report a statistic whose error is small compared with the difference it is being used to detect.

Both halves are load-bearing and the second is the one this collection got right by accident. Choosing the Kavrayskiy number as the ranking statistic was a decision about what distortion means — a root-mean-square of logarithms, symmetric between stretching and squeezing, which is what Chebyshev’s criterion is about — and it had nothing to do with quadrature. It happens also to be the statistic whose sampled version converges fastest, because averaging a bounded smooth function is the easiest thing a sample can be asked to do. The right statistic for the argument turned out to be the right statistic for the arithmetic, and there was no reason in advance for those to be the same.

That is a piece of luck this collection is entitled to record and not entitled to rely on. Which projection a weighting can make best shows how much freedom there already is in choosing a criterion; if a future criterion here is a maximum rather than a mean, everything in this anchor applies to it and none of this rung’s reassurance does.

Who found it, and when #

Nobody publishes this kind of audit, which is why it is here. A paper comparing map projections states its criterion and its region and does not state its sampling scheme, and a reader has no way to reconstruct one from the other. The convention is not carelessness: for the statistics that are usually reported — means over regions — the sampler genuinely does not matter, and a century of practice has established that by not producing visible contradictions.

The visible contradictions are all in the maxima, and they are in the literature. Published worst-case distortions for the same projection over the same region differ between sources by amounts that are usually attributed to different definitions of the region. Some of that difference is the region. Some of it is that one author sampled on a five-degree graticule and another on a one-degree one, and the second author’s number is larger for a reason that has nothing to do with either the map or the region.

Why the convention held for a century #

The defence of the practice is worth taking seriously, because it explains both why the omission is universal and why it is now worth repairing.

For the statistic that is usually reported, the sampler genuinely does not matter. A mean over a region converges for any reasonable sampler, the rate is good, and two authors using different schemes get numbers that agree to more digits than either would defend. A century of practice not producing contradictions is evidence, and it is evidence about means.

The contradictions are all in the extremes, and they are visible in the literature as unexplained disagreements between published worst-case figures for the same projection over the same region. Those are usually attributed to differing definitions of the region, which is a real cause and is not the only one — a denser sample finds a larger maximum, always, for reasons having nothing to do with the map.

So the convention is right about half of what it covers and silent about the other half, and there is nothing in a paper to say which half a given number belongs to. A reader cannot tell a mean, which is robust, from a maximum, which is not, by looking at how either is reported.

Which makes the repair narrower than the complaint suggests. Not every published number needs a sampler stated — only the extremes do, and they are a minority of what is published. One clause, on the numbers that need it, and the disagreements stop being mysterious.

Where this anchor goes next #

The five rungs here have audited the estimators this collection reads its numbers through: a maximum, a mean, a point set, a refinement, the derivative underneath all four, and the collection itself. What none of them has touched is the region. Every quantity in every rung is an average or an extreme over a stated region, and the regions in measure.js are boxes, caps and ellipses chosen for being easy to sample — a decision made once, for convenience, and inherited by every ranking since. Whether the verdicts survive being asked about a region with a real coastline’s shape is a different question from any asked here, and it is the one this anchor would take next.