Position digits, resolution and uncertainty

A position might appear as 60°00.000′ N or 60°00.0000′ N. The second notation distinguishes intervals ten times smaller, but does not demonstrate a measurement ten times better.

Digits tell us how a value was represented. To understand what it says, we also need its origin, time, datum and the uncertainty associated with the result.

This lesson develops that separation after the basic latitude and longitude formats.

Display resolution is the smallest step distinguished by the notation. With a 0.001-minute step and rounding to the nearest value, the difference between the value before rounding and the written value is at most half a step: 0.0005 minute.

This bounds representation error, not every position error. Truncating instead of rounding changes the interval; the conversion rule therefore matters.

For longitude, identical notation does not represent identical distance at every latitude. In a local spherical approximation, multiply equatorial distance by cos(latitude).

At 60°, cos(60°)=0.5: half a 0.001′ longitude step is approximately 0.463 m. Degrees therefore do not convert into metres through one universal constant.

Nor can the size of a latitude–longitude rectangle be directly equated with an uncertainty circle.

Imagine ten readings close together. Their small scatter describes repeatability under the observed conditions.

Yet all ten might share an eastward displacement caused by the same reference or influence. Averaging does not eliminate a common effect.

Accuracy concerns closeness to a reference value; repeatability concerns agreement between repeated results. Informal claims that something is “very precise” can conceal which property is actually being discussed.

An uncertainty needs to identify its quantity and evaluation method. An interval with a stated confidence level, a standard uncertainty and a maximum rounding bound are not equivalent expressions.

Half a step does not acquire 95% coverage simply because it is drawn around a point. Even “horizontal” leaves a question: does it describe each axis or radial distance?

A meaningful comparison retains the definition alongside the unit.

A wrong datum, mismatched time or coordinate copied with an incorrect sign belongs to another category again: it can displace the result without appearing in a small scatter of readings. The diagram shows a rounding grid, not the performance of an actual receiver.

The aim is understanding a value and its description; judging fitness for a manoeuvre requires separate applied assessment.