This article is from the Geographic Information Systems FAQ, by Lisa Nyman lnyman@census.gov with numerous contributions by others.
The historical definition of a "nautical mile" is "one minute of arc of a
great circle of the earth". Since the earth is not a perfect sphere, that
definition is ambiguous. However, the internationally accepted (SI) value
for the length of a nautical mile is (exactly, by definition) 1.852 km or
exactly 1.852/1.609344 international miles (that is, approximately 1.15078
miles - either "international" or "U.S. statute"). Thus, the implied
"official" circumference is 360 degrees times 60 minutes/degree times 1.852
km/minute = 40003.2 km. The implied radius is the circumference divided by
2 pi:
R = 6367 km = 3956 mi
The shape of the Earth is well approximated by an oblate spheroid with a
polar radius of 6357 km and an equatorial radius of 6378 km. PROVIDED a
spherical approximation is satisfactory, any value in that range will do,
such as
R (in km) = 6378 - 21 * sin(lat) See the WARNING below!
R (in mi) = 3963 - 13 * sin(lat)
where lat is a latitude near which the bulk of your calculations occur.
WARNING: This formula for R gives but a rough approximation to the radius
of curvature as a function of latitude. The radius of curvature varies
with direction and latitude; according to Snyder ("Map Projections - A
Working Manual", by John P. Snyder, U.S. Geological Survey Professional
Paper 1395, United States Government Printing Office, Washington DC, 1987,
p24), in the plane of the meridian it is given by
R' = a * (1 - e^2) / (1 - e^2 * sin^2(lat))^(3/2)
where a is the equatorial radius, b is the polar radius, and
e is the eccentricity of the ellipsoid = (1 - b^2/a^2)^(1/2).
 
Continue to: