This article is from the Photographic Lenses FAQ, by David Jacobson with numerous contributions by others.
The axis of the pivot should pass through the entrance pupil. The entrance pupil is the virtual image of the aperture as seen through the front of the lens.
When you've got it right, the entrance pupil will not shift relative to fixed objects as you rotate the camera. (Stop down the lens so you can see the diaphragm and aperture.)
There is a whole different type of panoramic camera in which the lens is rotated relative to the film. In this type of camera the lens rotates around the rear nodal point (for objects at infinity). See the lens tutorial for an explanation of nodal points.