A polygon on a sphere has no outside

Every essay on this ladder that has touched a polygon has quietly assumed the thing a polygon is for: that it separates an inside from an outside. Whether a point is inside depends on how the edges are read; computing its area needs a surface; a shared edge between two polygons is two edges. All of those take the inside for granted and argue about how to find it.

On a plane the assumption is a theorem. The Jordan curve theorem says a simple closed curve divides the plane into exactly two pieces, one bounded and one not, and the unbounded one is the outside. The word outside has a definition: the piece that reaches infinity.

A sphere has no infinity. A simple closed curve on it still divides it into exactly two pieces, and both of them are bounded. Neither piece is distinguished by anything intrinsic. Which one a file means is a convention, and the convention is not in the coordinates.

The two pieces, measured #

The area of a spherical polygon is a signed quantity. The standard formula sums a term over the ring’s edges and produces a number whose sign says which of the two pieces the ring names — specifically, the piece lying to the left of the direction of travel.

Take the modulus of that sum and the sign is gone, and with it the answer. That is what this site’s own area function does, and it has done it since the day it was written:

• for a ring round the pole at 80°, traversed eastwards, it returns 506.191 million square kilometres, which is everything except the cap;
• for the same ring traversed westwards, it returns 506.191 million;
• and the cap itself, 3.875 million square kilometres, is not what it returns in either direction.

That is not a bug in the sense of a wrong number. It is a decision, taken by whoever wrote the modulus, applied silently to every caller. The site’s cell machinery already noticed: cellArea in the cell library takes the branch smaller than a hemisphere, with a comment explaining that every cell there is far smaller than a hemisphere so the branch is unambiguous. That is a correct local fix and an exact statement of the problem — the branch is unambiguous for that caller, and the function has no way to know.

Four rules, and none of them is in the file #

Four rules are in general use and they are genuinely different:

Take the smaller piece. Simple, and right for anything smaller than a hemisphere, which is nearly everything. It fails for a ring at 60° south — the shape of any polygon drawn around the Antarctic coast — by a factor of 13.9, and it cannot decide at all for a ring near a great circle.

Take the piece the winding names. This is what RFC 7946 asks for: an exterior ring runs counterclockwise seen from outside, which is the same as saying the interior lies to the left of the direction of travel. It is unambiguous for every ring, and it depends entirely on a property of the file that most producers do not control and most consumers do not check.

Take the piece containing a stated point. Unambiguous and honest, and it requires a datum the format does not have.

Read the coordinates as a plane and take the bounded piece. This is what every planar geometry library does when handed longitude and latitude, and it is the only one of the four that is not a rule about the sphere at all. For a mid-latitude box it agrees with the others. For any of the rings that go all the way round, it returns an area of zero, because a ring of constant latitude spanning the full range of longitude is a straight horizontal line in the plane.

The ring for which no rule can decide #

The equator’s two pieces are 255.033 million square kilometres each, equal to six significant figures — they are exactly equal, and the residual is the discretisation of the ring. Any rule that prefers the smaller piece has nothing to prefer.

That is the worst case for a heuristic and it is not a contrived one: a ring that follows a great circle is what a satellite ground track, a hemisphere of illumination, or a great-circle corridor produces, and software that has been reliable for years on continental outlines meets it and produces an answer that is a coin toss.

Between the two extremes the ambiguity is a continuum. The mid-latitude box has a ratio of 291, the polar cap 131, the ring at 60° south 13.9, a ring at 20° north 2.0, and the equator 1.0. The heuristic’s reliability is exactly the inverse of that column, and nothing in a file says where on it a given ring sits.

Inside, when there is nowhere to fire the ray #

The classical point-in-polygon test fires a ray from the point to infinity and counts crossings: an odd count means inside. On a sphere there is no infinity, so the test becomes an arc from the point to a second point declared to be outside, and the answer is a statement about the pair.

Move the declaration from the south pole to the north and every point on the sphere changes its answer. That is not a numerical instability; it is the test working correctly and reporting that the two reference points are in different pieces.

For a ring that encloses neither pole, moving between the same two references changes nothing — zero of 400 points flip — which is why the problem is invisible in the ninety-nine per cent of cases where the polygon is a country and the implicit outside is wherever the coder was standing.

The test that is tempting and wrong #

Implementing the crossing test on a sphere has a trap in it worth recording, because it produced a confident wrong answer here before it was caught.

The natural test for whether two great-circle arcs cross is that each arc separates the other’s endpoints: the endpoints of one lie on opposite sides of the plane of the other, both ways round. That test is necessary and it is not sufficient. Two great circles meet at an antipodal pair of points, and the separation test is satisfied when one arc contains one of the two intersections and the other arc contains the other — which is not a crossing.

The symptom was a ray fired from a mid-latitude point to the south pole reporting a crossing of a ring around the north pole, at an edge on the far side of the world. Every count came out one too high for a family of points, and the resulting map of insides looked plausible.

The test that is right constructs the intersection explicitly and asks whether the same one lies inside both arcs, by checking that the two pieces it cuts each arc into add up to the whole arc. It is four lines longer and it does not have a family of cases where it is quietly wrong.

Why the common case hides it so well #

It is worth being precise about why software gets away with this, because the answer decides where to look for the failures.

Almost every polygon anybody stores is much smaller than a hemisphere. For such a ring the two pieces differ by a factor of hundreds, so the smaller-piece rule and the winding rule agree, the planar reading agrees, and any point a coder might declare to be outside is in the same piece as any other. Four rules, one answer, no disagreement to notice.

The rings that break it are a short and predictable list: anything containing a pole, anything within a few degrees of a great circle, and anything covering most of the world. Which is to say: Antarctica, the Arctic Ocean, satellite footprints, day-and-night terminators, ocean basins drawn as single polygons, and the “rest of the world” masks that thematic cartography uses to grey out everything a map is not about.

Every one of those is a case a general-purpose library will meet eventually and none of them is a case its test suite is likely to contain, because test suites are built from the data that motivated the library.

What was computed, and how #

Every ring is built from a stated rule — a parallel at a stated latitude, a box of stated corners, a meridian circle — and none of them is a country. That is deliberate: the whole subject is that a closed curve does not carry the information a country’s outline is assumed to carry, and a recognisable shape would smuggle that information back in through the reader.

The signed area is the standard sum with each longitude difference wrapped into (−π, π], which is the same wrap the antimeridian essay records as load-bearing: without it a ring encircling the globe accumulates a single −2π step on its closing segment, that step cancels the accumulated sum, and a hemisphere comes out with an area of exactly zero.

The check that makes the pair of numbers a measurement is that the two pieces of every ring account for the whole sphere, to nine figures, over five rings of very different shapes. If the formula were picking up a spurious multiple of 2π anywhere, that sum would not close.

And the refusal is the equator: two pieces equal to six figures, over a ring whose two pieces must be equal because it is a great circle. A formula that produced anything else there would be wrong in a way the polar cases could not reveal.

What the area of the world is for #

One number in the table above deserves a second look, because it is the one that makes the ambiguity concrete rather than topological.

The larger piece of the polar ring is 506.191 million square kilometres. The surface of the sphere is 510.066 million. So a program that hands back the wrong branch for a cap around the pole is not returning a number that is somewhat too large — it is returning the area of the Earth, to within the size of the cap it was asked about.

That is a failure with a useful property: it is enormous, and enormous failures get noticed. The dangerous branch errors are the ones near the middle of the ambiguity column, where the two answers differ by a factor of two or three and both look like the area of a plausible region. A ring at 20° north gives 167.8 and 342.3 million square kilometres, and neither number announces itself as wrong.

So the ambiguity is not uniformly dangerous. It is safe where it is huge and safe where it is tiny, and it is dangerous in a band in the middle, which is exactly where the ocean basins and the hemispheric footprints sit.

Where the dangerous band actually is #

The observation that the ambiguity is safe at both ends and dangerous in the middle deserves a boundary rather than a description, and the boundary is arithmetic.

Two answers are both plausible when they are within a small factor of each other — call it four, beyond which one of the two is obviously an area of the world and the other obviously is not. A ring whose smaller piece is a fraction p of the sphere has a ratio (1 − p)/p, so the dangerous band is p between 0.2 and 0.5: between a fifth and a half of the Earth’s surface, which is 102 to 255 million square kilometres.

For a cap that is an angular radius between 53.1° and 90°, since a cap of radius ρ covers (1 − cos ρ)/2 of the sphere. So the rings at risk are those reaching more than halfway from a pole to the equator and not much past the equator — which is a narrower and more specific description than ocean basins and footprints, and it can be checked on a ring before anything is computed.

One example is worth naming because it is drawn on maps daily. The ground footprint of a geostationary satellite is a cap of angular radius 81.3° — the arccosine of the Earth’s radius over the orbit’s — which covers 42.4 per cent of the sphere and has a piece ratio of 1.36. Every coverage map of a communications satellite, every weather-satellite disc, every broadcast footprint is a ring sitting almost exactly at the worst point of the ambiguity column, where the two candidate areas are 216 and 294 million square kilometres and neither looks wrong.

That is a better test than the heuristics the essay lists, and it costs nothing: compute both pieces and report the ratio. Where the ratio is above four, the smaller-piece rule is safe and can be applied with a note. Where it is under four, no rule that reads only the coordinates can decide, and the file has to be asked for its winding or the caller for a point. A library that returned an area together with the ratio of the two candidates would let every caller know which regime it was in, and no library does.

Where the model stops #

A ring is not a polygon. A real polygon has holes, and a real file has multi-polygons whose parts may be on opposite sides of the world. The winding rule extends to those — interior rings run the other way — and the extension inherits every ambiguity above and adds the question of which exterior ring a given interior one belongs to.

Self-intersecting rings have more than two pieces. Everything here is about simple closed curves. A ring that crosses itself divides the sphere into more pieces than two, and the signed sum returns a winding-number-weighted total that is not the area of any of them.

The measurements are on a sphere. On an ellipsoid the same statements hold with different numbers, and the area formula is the one the cell work uses, which differs from the spherical one by about a third of a per cent.

Nothing here is about validity. A file can be perfectly valid under every rule the specification states and still be ambiguous in the way this essay describes, because the specification’s own rule is about winding and most producers do not enforce it.

The generalisation #

The rule that comes out is uncomfortable and short: a ring plus a convention is a region; a ring alone is not. Any pipeline that reads polygons and does not state which convention it applies has an unstated parameter with two settings, and the setting only matters for the polygons that contain a pole or approach a hemisphere.

Which makes this a defect of exactly the shape the ladder’s earlier rungs keep finding: the information needed to interpret a file is not in the file, the default works for the common case, and the failure is silent and total rather than noisy and partial.

The practical advice is the one the format specification already gives and nobody enforces. Write the winding. Check it on read. And when a library returns an area, ask which of the two it returned — because on a sphere, unlike on a plane, that is a question with two possible answers and no default.

Who found it, and when #

The Jordan curve theorem is 1887, and the observation that its proof needs the unbounded component is in every topology course. That a sphere has no unbounded component is not a subtlety; it is the first thing anybody notices about a sphere.

The consequence for stored geometry surfaced with GeoJSON. The 2008 specification said nothing about winding; RFC 7946, published in 2016, added the right-hand rule for exterior rings and stated explicitly that a ring must not be assumed to enclose the smaller area. The rule exists because implementers had spent years discovering that Antarctica breaks everything, which it does for exactly the reason measured here.

Spherical geometry libraries diverged on the same question. Google’s S2 treats a loop as bounding the region to its left and says so; most planar libraries handed longitude and latitude take the bounded planar piece and say nothing. Both are defensible and they disagree on precisely the rings that matter.

Where the ladder goes next #

This ladder has now taken apart what a stored coordinate means, what a stored segment means, what an edge in a raster means, and what a stored ring means. What has not been examined is what happens when two files that disagree in any of those ways are overlaid — which is the operation every geographic analysis performs and the one where two independent sets of conventions meet.