7.8 Large Unix File Systems
Description
This article is from the storage FAQ part2, by Rodney D. Van Meter with numerous contributions by others.
7.8 Large Unix File Systems
There is now an industry group working on standardizing an API for
files larger than 2 GB (the max size normally supported on most Unix
systems). More info as I get it. The WWW-enabled can have a look at
http://www.sas.com:80/standards/large.file and see the various
proposals on the table.
Note that it is VERY easy to confuse whether an OS supports files
larger than 2 GB or file systems larger than 2 GB. My table lists
some of both (thanks to ben@rex.uokhsc.edu (Benjamin Z. Goldsteen),
Ed Hamrick (EdHamrick@aol.com) and Peter Poorman (poorman@convex.com)
for much of this information).
It is straightforward for systems with 64-bit integers to support
64-bit files; for systems with 32-bit integers it is more complex. On
most 32-bit systems the offsets passed around inside the kernel (most
importantly, at the VFS layer) the file offsets and sizes tend to be
passed as 32-bit (signed) integers, meaning no files >2^31.
On most systems, the argument to lseek is of type off_t, which (on
SunOS and Linux, and plausibly on OSF/1 and others) is declared in a
header file as "typedef long off_t;".
For clients to really have access to large files, three pieces are
required: local FS support, an appropriate network protocol, and
server support for 64-bit FSes. For FTP access, I believe literally
inifinitely large files are possible, but I'm not sure(SHMO). For NFS
access, NFS V2 supports only 2GB files. NFS V3, just becoming
available now, supports full 64-bit files, I believe (anybody have a
reference to the docs? RFC? SHMO). With the notable exception of
Unitree (which does not use, depend on, or appear as, a local FS on
the server), server support for 64-bit files is provided only when the
server's own local FSes are 64-bit.
Even for the systems that do support large files, not all are
programmer or user-transparent for supporting large files. UniCOS is,
OSF/1 is, ConvexOS is not (there are two system calls, lseek and
lseek64, with 32-bit and 64-bit file offsets, respectively, though the
Fortran interface is transparent).
This brings up the related issues. A complete large files implementation
needs not only the system calls, but also the stdio library and the runtime
libraries for the languages (Fortran, Cobol,...). Further, system utilities
(sed, dd, etcetera) need to be capable of dealing with large files.
(It has been pointed out that the GNU C compiler runs on most of these
machines, so it is possible to use "long long" as a 64-bit int on
them, but what matters for file systems is the system compiler.)
Here's the start of a table on these. Really such a simple table can't
do the problem justice, but it'll give you an idea. Keep in mind that
many of these systems support many file system types; I've listed only
the most interesting so far from this point of view. I'd like to flesh
it out more completely, though.
1 GB = 2^30, 1 TB = 2^40, 1 PB = 2^50, 1 EB = 2^60
NYR = Not Yet Released
OS/hardware 64-bit C max max NFS info
datatype par- file V3 updated
tition size sup
size (bytes)
UniCOS (Cray vector) int, long ? 8 EB? ? 8/94
ConvexOS long long 1 TB 1 TB N 9/94
Alpha AXP OSF/1 V3.0
AFS long 128 GB 16 TB 8/94 9/94
Paragon OSF/1 ? 8 EB* 8 EB N 2/95
UTS (Amdahl) ? ? 8 EB? ? 8/94
HP/UX 9 (HP 9xxx) ? 4 GB ? ? 8/94
Silicon Graphics
IRIX 5.2 EFS long long 8 GB 2 GB N 9/94
IRIX 6.0 EFS long 8 GB 2 GB N 9/94 (NYR)
IRIX 5.3 XFS ?long long? ? ?TB? Y 9/94 (NYR)
AIX (IBM RS/6000)
4.1 JFS long long 64 GB 2 GB N 8/94
Solaris 2.x (Sun Sparc) long long 1 TB 2 GB (soon?) 9/94
BSD 4.4 long long ? 8 EB? ? 8/94
Linux long long 1 TB 2 GB ? 9/94
DG/UX 5.4 long long 2 TB 2 GB ? 9/94
Alliant Concentrix long long ?>2 GB ?>2 GB N 9/94 (dead)
* The Paragon PFS (Parallel File System), as I understand it, parallelizes
access to the files; each partition striped across is limited to 2GB, so
really the max partition size is 2GB * # of disks that can be attached.
A slightly more detailed
description of certain implementations
is available with the WWW version.
In addition, the HPSS (see above) supports large files, as does
Unitree (though the Unitree interface to them is limited).
Continue to:
My Books
