List Mode Format Specifications¶

The manual page for spax is not yet ready. The following text is a quotation from the POSIX.1-2001 standard.

In list mode with the -o listopt=format option, the format argument shall be applied for each selected file. The pax utility shall append a <newline> to the listopt output for each selected file. The format argument shall be used as the format string described in the Base Definitions volume of IEEE Std 1003.1-2001, Chapter 5, File Format Notation, with the exceptions 1. through 5. defined in the EXTENDED DESCRIPTION section of printf(3), plus the following exceptions:

6.

The sequence (keyword) can occur before a format conversion specifier. The conversion argument is defined by the value of keyword. The implementation shall support the following keywords:

7.

An additional conversion specifier character, T, shall be used to specify time formats. The T conversion specifier character can be preceded by the sequence (keyword=subformat), where subformat is a date format as defined by date operands. The default keyword shall be mtime and the default subformat shall be:

%b %e %H:%M %Y

8.

An additional conversion specifier character, M, shall be used to specify the file mode string as defined in ls(1) Standard Output. If (keyword) is omitted, the mode keyword shall be used. For example, %.1M writes the single character corresponding to the <entry type> field of the ls -l command.

9.

An additional conversion specifier character, D, shall be used to specify the device for block or special files, if applicable, in an implementation-defined format. If not applicable, and (keyword) is specified, then this conversion shall be equivalent to %(keyword)u. If not applicable, and (keyword) is omitted, then this conversion shall be equivalent to <space>.

10.

An additional conversion specifier character, F, shall be used to specify a pathname. The F conversion character can be preceded by a sequence of comma-separated keywords:

(keyword[,keyword] ... )

11.

An additional conversion specifier character, L, shall be used to specify a symbolic line expansion. If the current file is a symbolic link, then %L shall expand to:

"%s -> %s", <value of keyword>, <contents of link>

Otherwise, the %L conversion specification shall be the equivalent of %F.

pax Interchange Format¶

A pax archive tape or file produced in the -x pax format shall contain a series of blocks. The physical layout of the archive shall be identical to the ustar format described in ustar Interchange Format. Each file archived shall be represented by the following sequence:

At the end of the archive file there shall be two 512-byte blocks filled with binary zeros, interpreted as an end-of-archive indicator.

A schematic of an example archive with global extended header records and two actual files is shown in pax Format Archive Example. In the example, the second file in the archive has no extended header preceding it, presumably because it has no need for extended attributes.

Figure: pax Format Archive Example

pax Header Block¶

The pax header block shall be identical to the ustar header block described in ustar Interchange Format, except that two additional typeflag values are defined:

x

Represents extended header records for the following file in the archive (which shall have its own ustar header block). The format of these extended header records shall be as described in pax Extended Header.

g

Represents global extended header records for the following files in the archive. The format of these extended header records shall be as described in pax Extended Header. Each value shall affect all subsequent files that do not override that value in their own extended header record and until another global extended header record is reached that provides another value for the same field. The typeflag g global headers should not be used with interchange media that could suffer partial data loss in transporting the archive.

For both of these types, the size field shall be the size of the extended header records in octets. The other fields in the header block are not meaningful to this version of the pax utility. However, if this archive is read by a pax utility conforming to the ISO POSIX-2:1993 standard, the header block fields are used to create a regular file that contains the extended header records as data. Therefore, header block field values should be selected to provide reasonable file access to this regular file.

A further difference from the ustar header block is that data blocks for files of typeflag 1 (the digit one) (hard link) may be included, which means that the size field may be greater than zero. Archives created by pax -o linkdata shall include these data blocks with the hard links.

pax Extended Header¶

A pax extended header contains values that are inappropriate for the ustar header block because of limitations in that format: fields requiring a character encoding other than that described in the ISO/IEC 646:1991 standard, fields representing file attributes not described in the ustar header, and fields whose format or length do not fit the requirements of the ustar header. The values in an extended header add attributes to the following file (or files; see the description of the typeflag g header block) or override values in the following header block(s), as indicated in the following list of keywords.

An extended header shall consist of one or more records, each constructed as follows:

"%d %s=%s\n", <length>, <keyword>, <value>

The extended header records shall be encoded according to the ISO/IEC 10646-1:2000 standard (UTF-8). The <length> field, <blank>, equals sign, and <newline> shown shall be limited to the portable character set, as encoded in UTF-8. The <keyword> and <value> fields can be any UTF-8 characters. The <length> field shall be the decimal length of the extended header record in octets, including the trailing <newline>.

The <keyword> field shall be one of the entries from the following list or a keyword provided as an implementation extension. Keywords consisting entirely of lowercase letters, digits, and periods are reserved for future standardization. A keyword shall not include an equals sign. (In the following list, the notations "file(s)" or "block(s)" is used to acknowledge that a keyword affects the following single file after a typeflag x extended header, but possibly multiple files after typeflag g. Any requirements in the list for pax to include a record when in write or copy mode shall apply only when such a record has not already been provided through the use of the -o option. When used in copy mode, pax shall behave as if an archive had been created with applicable extended header records and then extracted.)

atime

The file access time for the following file(s), equivalent to the value of the st_atime member of the stat structure for a file, as described by the stat(2) function. The access time shall be restored if the process has the appropriate privilege required to do so. The format of the <value> shall be as described in pax Extended Header File Times.

charset

The name of the character set used to encode the data in the following file(s). The entries in the following table are defined to refer to known standards; additional names may be agreed on between the originator and recipient.

The encoding is included in an extended header for information only; when pax is used as described in IEEE Std 1003.1-2001, it shall not translate the file data into any other encoding. The BINARY entry indicates unencoded binary data.

When used in write or copy mode, it is implementation-defined whether pax includes a charset extended header record for a file.

comment

A series of characters used as a comment. All characters in the <value> field shall be ignored by pax.

gid

The group ID of the group that owns the file, expressed as a decimal number using digits from the ISO/IEC 646:1991 standard. This record shall override the gid field in the following header block(s). When used in write or copy mode, pax shall include a gid extended header record for each file whose group ID is greater than 2097151 (octal 7777777).

gname

The group of the file(s), formatted as a group name in the group database. This record shall override the gid and gname fields in the following header block(s), and any gid extended header record. When used in read, copy, or list mode, pax shall translate the name from the UTF-8 encoding in the header record to the character set appropriate for the group database on the receiving system. If any of the UTF-8 characters cannot be translated, and if the -o invalid=UTF-8 option is not specified, the results are implementation-defined. When used in write or copy mode, pax shall include a gname extended header record for each file whose group name cannot be represented entirely with the letters and digits of the portable character set.

linkpath

The pathname of a link being created to another file, of any type, previously archived. This record shall override the linkname field in the following ustar header block(s). The following ustar header block shall determine the type of link created. If typeflag of the following header block is 1, it shall be a hard link. If typeflag is 2, it shall be a symbolic link and the linkpath value shall be the contents of the symbolic link. The pax utility shall translate the name of the link (contents of the symbolic link) from the UTF-8 encoding to the character set appropriate for the local file system. When used in write or copy mode, pax shall include a linkpath extended header record for each link whose pathname cannot be represented entirely with the members of the portable character set other than NUL.

mtime

The file modification time of the following file(s), equivalent to the value of the st_mtime member of the stat structure for a file, as described in the stat(2) function. This record shall override the mtime field in the following header block(s). The modification time shall be restored if the process has the appropriate privilege required to do so. The format of the <value> shall be as described in pax Extended Header File Times.

path

The pathname of the following file(s). This record shall override the name and prefix fields in the following header block(s). The pax utility shall translate the pathname of the file from the UTF-8 encoding to the character set appropriate for the local file system.

When used in write or copy mode, pax shall include a path extended header record for each file whose pathname cannot be represented entirely with the members of the portable character set other than NUL.

realtime.any

The keywords prefixed by "realtime." are reserved for future standardization.

security.any

The keywords prefixed by "security." are reserved for future standardization.

size

The size of the file in octets, expressed as a decimal number using digits from the ISO/IEC 646:1991 standard. This record shall override the size field in the following header block(s). When used in write or copy mode, pax shall include a size extended header record for each file with a size value greater than 8589934591 (octal 77777777777).

uid

The user ID of the file owner, expressed as a decimal number using digits from the ISO/IEC 646:1991 standard. This record shall override the uid field in the following header block(s). When used in write or copy mode, pax shall include a uid extended header record for each file whose owner ID is greater than 2097151 (octal 7777777).

uname

The owner of the following file(s), formatted as a user name in the user database. This record shall override the uid and uname fields in the following header block(s), and any uid extended header record. When used in read, copy, or list mode, pax shall translate the name from the UTF-8 encoding in the header record to the character set appropriate for the user database on the receiving system. If any of the UTF-8 characters cannot be translated, and if the -o invalid=UTF-8 option is not specified, the results are implementation-defined. When used in write or copy mode, pax shall include a uname extended header record for each file whose user name cannot be represented entirely with the letters and digits of the portable character set.

If the <value> field is zero length, it shall delete any header block field, previously entered extended header value, or global extended header value of the same name.

If a keyword in an extended header record (or in a -o option-argument) overrides or deletes a corresponding field in the ustar header block, pax shall ignore the contents of that header block field.

Unlike the ustar header block fields, NULs shall not delimit <value>s; all characters within the <value> field shall be considered data for the field. None of the length limitations of the ustar header block fields in ustar Header Block shall apply to the extended header records.

pax Extended Header Keyword Precedence¶

This section describes the precedence in which the various header records and fields and command line options are selected to apply to a file in the archive. When pax is used in read or list modes, it shall determine a file attribute in the following sequence:

pax Extended Header File Times¶

The pax utility shall write an mtime record for each file in write or copy modes if the file's modification time cannot be represented exactly in the ustar header logical record described in ustar Interchange Format. This can occur if the time is out of ustar range, or if the file system of the underlying implementation supports non-integer time granularities and the time is not an integer. All of these time records shall be formatted as a decimal representation of the time in seconds since the Epoch. If a period ('.') decimal point character is present, the digits to the right of the point shall represent the units of a subsecond timing granularity, where the first digit is tenths of a second and each subsequent digit is a tenth of the previous digit. In read or copy mode, the pax utility shall truncate the time of a file to the greatest value that is not greater than the input header file time. In write or copy mode, the pax utility shall output a time exactly if it can be represented exactly as a decimal number, and otherwise shall generate only enough digits so that the same time shall be recovered if the file is extracted on a system whose underlying implementation supports the same time granularity.

ustar Interchange Format¶

A ustar archive tape or file shall contain a series of logical records. Each logical record shall be a fixed-size logical record of 512 octets (see below). Although this format may be thought of as being stored on 9-track industry-standard 12.7 mm (0.5 in) magnetic tape, other types of transportable media are not excluded. Each file archived shall be represented by a header logical record that describes the file, followed by zero or more logical records that give the contents of the file. At the end of the archive file there shall be two 512-octet logical records filled with binary zeros, interpreted as an end-of-archive indicator.

The logical records may be grouped for physical I/O operations, as described under the -b blocksize and -x ustar options. Each group of logical records may be written with a single operation equivalent to the write(2) function. On magnetic tape, the result of this write shall be a single tape physical block. The last physical block shall always be the full size, so logical records after the two zero logical records may contain undefined data.

The header logical record shall be structured as shown in the following table. All lengths and offsets are in decimal.

Table: ustar Header Block

All characters in the header logical record shall be represented in the coded character set of the ISO/IEC 646:1991 standard. For maximum portability between implementations, names should be selected from characters represented by the portable filename character set as octets with the most significant bit zero. If an implementation supports the use of characters outside of slash and the portable filename character set in names for files, users, and groups, one or more implementation-defined encodings of these characters shall be provided for interchange purposes.

However, the pax utility shall never create filenames on the local system that cannot be accessed via the procedures described in IEEE Std 1003.1-2001. If a filename is found on the medium that would create an invalid filename, it is implementation-defined whether the data from the file is stored on the file hierarchy and under what name it is stored. The pax utility may choose to ignore these files as long as it produces an error indicating that the file is being ignored.

Each field within the header logical record is contiguous; that is, there is no padding used. Each character on the archive medium shall be stored contiguously.

The fields magic, uname, and gname are character strings each terminated by a NUL character. The fields name, linkname, and prefix are NUL-terminated character strings except when all characters in the array contain non-NUL characters including the last character. The version field is two octets containing the characters "00" (zero-zero). The typeflag contains a single character. All other fields are leading zero-filled octal numbers using digits from the ISO/IEC 646:1991 standard IRV. Each numeric field is terminated by one or more <space> or NUL characters.

The name and the prefix fields shall produce the pathname of the file. A new pathname shall be formed, if prefix is not an empty string (its first character is not NUL), by concatenating prefix (up to the first NUL character), a slash character, and name; otherwise, name is used alone. In either case, name is terminated at the first NUL character. If prefix begins with a NUL character, it shall be ignored. In this manner, pathnames of at most 256 characters can be supported. If a pathname does not fit in the space provided, pax shall notify the user of the error, and shall not store any part of the file-header or data-on the medium.

The linkname field, described below, shall not use the prefix to produce a pathname. As such, a linkname is limited to 100 characters. If the name does not fit in the space provided, pax shall notify the user of the error, and shall not attempt to store the link on the medium.

The mode field provides 12 bits encoded in the ISO/IEC 646:1991 standard octal digit representation. The encoded bits shall represent the following values:

Table: ustar mode Field

When appropriate privilege is required to set one of these mode bits, and the user restoring the files from the archive does not have the appropriate privilege, the mode bits for which the user does not have appropriate privilege shall be ignored. Some of the mode bits in the archive format are not mentioned elsewhere in this volume of IEEE Std 1003.1-2001. If the implementation does not support those bits, they may be ignored.

The uid and gid fields are the user and group ID of the owner and group of the file, respectively.

The size field is the size of the file in octets. If the typeflag field is set to specify a file to be of type 1 (a link) or 2 (a symbolic link), the size field shall be specified as zero. If the typeflag field is set to specify a file of type 5 (directory), the size field shall be interpreted as described under the definition of that record type. No data logical records are stored for types 1, 2, or 5. If the typeflag field is set to 3 (character special file), 4 (block special file), or 6 (FIFO), the meaning of the size field is unspecified by this volume of IEEE Std 1003.1-2001, and no data logical records shall be stored on the medium. Additionally, for type 6, the size field shall be ignored when reading. If the typeflag field is set to any other value, the number of logical records written following the header shall be (size+511)/512, ignoring any fraction in the result of the division.

The mtime field shall be the modification time of the file at the time it was archived. It is the ISO/IEC 646:1991 standard representation of the octal value of the modification time obtained from the stat(2) function.

The chksum field shall be the ISO/IEC 646:1991 standard IRV representation of the octal value of the simple sum of all octets in the header logical record. Each octet in the header shall be treated as an unsigned value. These values shall be added to an unsigned integer, initialized to zero, the precision of which is not less than 17 bits. When calculating the checksum, the chksum field is treated as if it were all spaces.

The typeflag field specifies the type of file archived. If a particular implementation does not recognize the type, or the user does not have appropriate privilege to create that type, the file shall be extracted as if it were a regular file if the file type is defined to have a meaning for the size field that could cause data logical records to be written on the medium (see the previous description for size). If conversion to a regular file occurs, the pax utility shall produce an error indicating that the conversion took place. All of the typeflag fields shall be coded in the ISO/IEC 646:1991 standard IRV:

0

Represents a regular file. For backwards-compatibility, a typeflag value of binary zero ('\0') should be recognized as meaning a regular file when extracting files from the archive. Archives written with this version of the archive file format create regular files with a typefla value of the ISO/IEC 646:1991 standard IRV '0'.

1

Represents a file linked to another file, of any type, previously archived. Such files are identified by having the same device and file serial numbers, and pathnames that refer to different directory entries. All such files shall be archived as linked files. The linked-to name is specified in the linkname field with a NUL-character terminator if it is less than 100 octets in length.

2

Represents a symbolic link. The contents of the symbolic link shall be stored in the linkname field.

3,4

Represent character special files and block special files respectively. In this case the devmajor and devminor fields shall contain information defining the device, the format of which is unspecified by this volume of IEEE Std 1003.1-2001. Implementations may map the device specifications to their own local specification or may ignore the entry.

5

Specifies a directory or subdirectory. On systems where disk allocation is performed on a directory basis, the size field shall contain the maximum number of octets (which may be rounded to the nearest disk block allocation unit) that the directory may hold. A size field of zero indicates no such limiting. Systems that do not support limiting in this manner should ignore the size field.

6

Specifies a FIFO special file. Note that the archiving of a FIFO file archives the existence of this file and not its contents.

7

Reserved to represent a file to which an implementation has associated some high-performance attribute. Implementations without such extensions should treat this file as a regular file (type 0).

A-Z

The letters 'A' to 'Z', inclusive, are reserved for custom implementations. All other values are reserved for future versions of IEEE Std 1003.1-2001.

It is unspecified whether files with pathnames that refer to the same directory entry are archived as linked files or as separate files. If they are archived as linked files, this means that attempting to extract both pathnames from the resulting archive will always cause an error (unless the -u option is used) because the link cannot be created.

It is unspecified whether files with the same device and file serial numbers being appended to an archive are treated as linked files to members that were in the archive before the append.

Attempts to archive a socket using ustar interchange format shall produce a diagnostic message. Handling of other file types is implementation-defined.

The magic field is the specification that this archive was output in this archive format. If this field contains ustar (the five characters from the ISO/IEC 646:1991 standard IRV shown followed by NUL), the uname and gname fields shall contain the ISO/IEC 646:1991 standard IRV representation of the owner and group of the file, respectively (truncated to fit, if necessary). When the file is restored by a privileged, protection-preserving version of the utility, the user and group databases shall be scanned for these names. If found, the user and group IDs contained within these files shall be used rather than the values contained within the uid and gid fields.

cpio Interchange Format¶

The octet-oriented cpio archive format shall be a series of entries, each comprising a header that describes the file, the name of the file, and then the contents of the file.

An archive may be recorded as a series of fixed-size blocks of octets. This blocking shall be used only to make physical I/O more efficient. The last group of blocks shall always be at the full size.

For the octet-oriented cpio archive format, the individual entry information shall be in the order indicated and described by the following table; see also the <cpio.h> header.

Table: Octet-Oriented cpio Archive Entry

cpio Header¶

For each file in the archive, a header as defined previously shall be written. The information in the header fields is written as streams of the ISO/IEC 646:1991 standard characters interpreted as octal numbers. The octal numbers shall be extended to the necessary length by appending the ISO/IEC 646:1991 standard IRV zeros at the most-significant-digit end of the number; the result is written to the most-significant digit of the stream of octets first. The fields shall be interpreted as follows:

c_magic

Identify the archive as being a transportable archive by containing the identifying value "070707".

c_dev, c_ino

Contains values that uniquely identify the file within the archive (that is, no files contain the same pair of c_dev and c_ino values unless they are links to the same file). The values shall be determined in an unspecified manner.

c_mode

Contains the file type and access permissions as defined in the following table.

c_uid

Contains the user ID of the owner.

c_gid

Contains the group ID of the group.

c_nlink

Contains a number greater than or equal to the number of links in the archive referencing the file. If the -a option is used to append to a cpio archive, then the pax utility need not account for the files in the existing part of the archive when calculating the c_nlink values for the appended part of the archive, and need not alter the c_nlink values in the existing part of the archive if additional files with the same c_dev and c_ino values are appended to the archive.

c_rdev

Contains implementation-defined information for character or block special files.

c_mtime

Contains the latest time of modification of the file at the time the archive was created.

c_namesize

Contains the length of the pathname, including the terminating NUL character.

c_filesize

Contains the length of the file in octets. This shall be the length of the data section following the header structure.

cpio Filename¶

The c_name field shall contain the pathname of the file. The length of this field in octets is the value of c_namesize.

If a filename is found on the medium that would create an invalid pathname, it is implementation-defined whether the data from the file is stored on the file hierarchy and under what name it is stored.

All characters shall be represented in the ISO/IEC 646:1991 standard IRV. For maximum portability between implementations, names should be selected from characters represented by the portable filename character set as octets with the most significant bit zero. If an implementation supports the use of characters outside the portable filename character set in names for files, users, and groups, one or more implementation-defined encodings of these characters shall be provided for interchange purposes. However, the pax utility shall never create filenames on the local system that cannot be accessed via the procedures described previously in this volume of IEEE Std 1003.1-2001. If a filename is found on the medium that would create an invalid filename, it is implementation-defined whether the data from the file is stored on the local file system and under what name it is stored. The pax utility may choose to ignore these files as long as it produces an error indicating that the file is being ignored.

cpio File Data¶

Following c_name, there shall be c_filesize octets of data. Interpretation of such data occurs in a manner dependent on the file. If c_filesize is zero, no data shall be contained in c_filedata.

When restoring from an archive:

•

If the user does not have the appropriate privilege to create a file of the specified type, pax shall ignore the entry and write an error message to standard error.

•

Only regular files have data to be restored. Presuming a regular file meets any selection criteria that might be imposed on the format-reading utility by the user, such data shall be restored.

•

If a user does not have appropriate privilege to set a particular mode flag, the flag shall be ignored. Some of the mode flags in the archive format are not mentioned elsewhere in this volume of IEEE Std 1003.1-2001. If the implementation does not support those flags, they may be ignored.

cpio Special Entries¶

FIFO special files, directories, and the trailer shall be recorded with c_filesize equal to zero. For other special files, c_filesize is unspecified by this volume of IEEE Std 1003.1-2001. The header for the next file entry in the archive shall be written directly after the last octet of the file entry preceding it. A header denoting the filename TRAILER!!! shall indicate the end of the archive; the contents of octets in the last block of the archive following such a header are undefined.

pax Interchange Format¶

The new POSIX data interchange format was developed primarily to satisfy international concerns that the ustar and cpio formats did not provide for file, user, and group names encoded in characters outside a subset of the ISO/IEC 646:1991 standard. The standard developers realized that this new POSIX data interchange format should be very extensible because there were other requirements they foresaw in the near future:

•

Support international character encodings and locale information

•

Support security information (ACLs, and so on)

•

Support future file types, such as realtime or contiguous files

•

Include data areas for implementation use

•

Support systems with words larger than 32 bits and timers with subsecond granularity

The following were not goals for this format because these are better handled by separate utilities or are inappropriate for a portable format:

•

Encryption

•

Compression

•

Data translation between locales and codesets

•

inode storage

The format chosen to support the goals is an extension of the ustar format. Of the two formats previously available, only the ustar format was selected for extensions because:

•

It was easier to extend in an upwards-compatible way. It offered version flags and header block type fields with room for future standardization. The cpio format, while possessing a more flexible file naming methodology, could not be extended without breaking some theoretical implementation or using a dummy filename that could be a legitimate filename.

•

Industry experience since the original "tar wars" fought in developing the ISO POSIX-1 standard has clearly been in favor of the ustar format, which is generally the default output format selected for pax implementations on new systems.

The new format was designed with one additional goal in mind: reasonable behavior when an older tar or pax utility happened to read an archive. Since the POSIX.1-1990 standard mandated that a "format-reading utility" had to treat unrecognized typeflag values as regular files, this allowed the format to include all the extended information in a pseudo-regular file that preceded each real file. An option is given that allows the archive creator to set up reasonable names for these files on the older systems. Also, the normative text suggests that reasonable file access values be used for this ustar header block. Making these header files inaccessible for convenient reading and deleting would not be reasonable. File permissions of 600 or 700 are suggested.

The ustar typeflag field was used to accommodate the additional functionality of the new format rather than magic or version because the POSIX.1-1990 standard (and, by reference, the previous version of pax), mandated the behavior of the format-reading utility when it encountered an unknown typeflag, but was silent about the other two fields.

Early proposals of the first revision to IEEE Std 1003.1-2001 contained a proposed archive format that was based on compatibility with the standard for tape files (ISO 1001, similar to the format used historically on many mainframes and minicomputers). This format was overly complex and required considerable overhead in volume and header records. Furthermore, the standard developers felt that it would not be acceptable to the community of POSIX developers, so it was later changed to be a format more closely related to historical practice on POSIX systems.

The prefix and name split of pathnames in ustar was replaced by the single path extended header record for simplicity.

The concept of a global extended header (typeflag g) was controversial. If this were applied to an archive being recorded on magnetic tape, a few unreadable blocks at the beginning of the tape could be a serious problem; a utility attempting to extract as many files as possible from a damaged archive could lose a large percentage of file header information in this case. However, if the archive were on a reliable medium, such as a CD-ROM, the global extended header offers considerable potential size reductions by eliminating redundant information. Thus, the text warns against using the global method for unreliable media and provides a method for implanting global information in the extended header for each file, rather than in the typeflag g records.

No facility for data translation or filtering on a per-file basis is included because the standard developers could not invent an interface that would allow this in an efficient manner. If a filter, such as encryption or compression, is to be applied to all the files, it is more efficient to apply the filter to the entire archive as a single file. The standard developers considered interfaces that would invoke a shell script for each file going into or out of the archive, but the system overhead in this approach was considered to be too high.

One such approach would be to have filter= records that give a pathname for an executable. When the program is invoked, the file and archive would be open for standard input/output and all the header fields would be available as environment variables or command-line arguments. The standard developers did discuss such schemes, but they were omitted from IEEE Std 1003.1-2001 due to concerns about excessive overhead. Also, the program itself would need to be in the archive if it were to be used portably.

There is currently no portable means of identifying the character set(s) used for a file in the file system. Therefore, pax has not been given a mechanism to generate charset records automatically. The only portable means of doing this is for the user to write the archive using the -o charset=string command line option. This assumes that all of the files in the archive use the same encoding. The "implementation-defined" text is included to allow for a system that can identify the encodings used for each of its files.

The table of standards that accompanies the charset record description is acknowledged to be very limited. Only a limited number of character set standards is reasonable for maximal interchange. Any character set is, of course, possible by prior agreement. It was suggested that EBCDIC be listed, but it was omitted because it is not defined by a formal standard. Formal standards, and then only those with reasonably large followings, can be included here, simply as a matter of practicality. The <value>s represent names of officially registered character sets in the format required by the ISO 2375:1985 standard.

The normal comma or <blank>-separated list rules are not followed in the case of keyword options to allow ease of argument parsing for getopts.

Further information on character encodings is in pax Archive Character Set Encoding/Decoding.

The standard developers have reserved keyword name space for vendor extensions. It is suggested that the format to be used is:

VENDOR.keyword

where VENDOR is the name of the vendor or organization in all uppercase letters. It is further suggested that the keyword following the period be named differently than any of the standard keywords so that it could be used for future standardization, if appropriate, by omitting the VENDOR prefix.

The <length> field in the extended header record was included to make it simpler to step through the records, even if a record contains an unknown format (to a particular pax) with complex interactions of special characters. It also provides a minor integrity checkpoint within the records to aid a program attempting to recover files from a damaged archive.

There are no extended header versions of the devmajor and devminor fields because the unspecified format ustar header field should be sufficient. If they are not, vendor-specific extended keywords (such as VENDOR.devmajor) should be used.

Device and i-number labeling of files was not adopted from cpio; files are interchanged strictly on a symbolic name basis, as in ustar.

Just as with the ustar format descriptions, the new format makes no special arrangements for multi-volume archives. Each of the pax archive types is assumed to be inside a single POSIX file and splitting that file over multiple volumes (diskettes, tape cartridges, and so on), processing their labels, and mounting each in the proper sequence are considered to be implementation details that cannot be described portably.

The pax format is intended for interchange, not only for backup on a single (family of) systems. It is not as densely packed as might be possible for backup:

•

It contains information as coded characters that could be coded in binary.

•

It identifies extended records with name fields that could be omitted in favor of a fixed-field layout.

•

It translates names into a portable character set and identifies locale-related information, both of which are probably unnecessary for backup.

The requirements on restoring from an archive are slightly different from the historical wording, allowing for non-monolithic privilege to bring forward as much as possible. In particular, attributes such as "high performance file" might be broadly but not universally granted while set-user-ID or chown(2) might be much more restricted. There is no implication in IEEE Std 1003.1-2001 that the security information be honored after it is restored to the file hierarchy, in spite of what might be improperly inferred by the silence on that topic. That is a topic for another standard.

Links are recorded in the fashion described here because a link can be to any file type. It is desirable in general to be able to restore part of an archive selectively and restore all of those files completely. If the data is not associated with each link, it is not possible to do this. However, the data associated with a file can be large, and when selective restoration is not needed, this can be a significant burden. The archive is structured so that files that have no associated data can always be restored by the name of any link name of any link, and the user may choose whether data is recorded with each instance of a file that contains data. The format permits mixing of both types of links in a single archive; this can be done for special needs, and pax is expected to interpret such archives on input properly, despite the fact that there is no pax option that would force this mixed case on output. (When -o linkdata is used, the output must contain the duplicate data, but the implementation is free to include it or omit it when -o linkdata is not used.)

The time values are included as extended header records for those implementations needing more than the eleven octal digits allowed by the ustar format. Portable file timestamps cannot be negative. If pax encounters a file with a negative timestamp in copy or write mode, it can reject the file, substitute a non-negative timestamp, or generate a non-portable timestamp with a leading '-'. Even though some implementations can support finer file-time granularities than seconds, the normative text requires support only for seconds since the Epoch because the ISO POSIX-1 standard states them that way. The ustar format includes only mtime; the new format adds atime and ctime for symmetry. The atime access time restored to the file system will be affected by the -p a and -p e options. The ctime creation time (actually inode modification time) is described with "appropriate privilege" so that it can be ignored when writing to the file system. POSIX does not provide a portable means to change file creation time. Nothing is intended to prevent a non-portable implementation of pax from restoring the value.

The gid, size, and uid extended header records were included to allow expansion beyond the sizes specified in the regular tar header. New file system architectures are emerging that will exhaust the 12-digit size field. There are probably not many systems requiring more than 8 digits for user and group IDs, but the extended header values were included for completeness, allowing overrides for all of the decimal values in the tar header.

The standard developers intended to describe the effective results of pax with regard to file ownerships and permissions; implementations are not restricted in timing or sequencing the restoration of such, provided the results are as specified.

Much of the text describing the extended headers refers to use in "write or copy modes". The copy mode references are due to the normative text: "The effect of the copy shall be as if the copied files were written to an archive file and then subsequently extracted ...". There is certainly no way to test whether pax is actually generating the extended headers in copy mode, but the effects must be as if it had.

pax Archive Character Set Encoding/Decoding¶

There is a need to exchange archives of files between systems of different native codesets. Filenames, group names, and user names must be preserved to the fullest extent possible when an archive is read on the receiving platform. Translation of the contents of files is not within the scope of the pax utility.

There will also be the need to represent characters that are not available on the receiving platform. These unsupported characters cannot be automatically folded to the local set of characters due to the chance of collisions. This could result in overwriting previous extracted files from the archive or pre-existing files on the system.

For these reasons, the codeset used to represent characters within the extended header records of the pax archive must be sufficiently rich to handle all commonly used character sets. The fields requiring translation include, at a minimum, filenames, user names, group names, and link pathnames. Implementations may wish to have localized extended keywords that use non-portable characters.

The standard developers considered the following options:

•

The archive creator specifies the well-defined name of the source codeset. The receiver must then recognize the codeset name and perform the appropriate translations to the destination codeset.

•

The archive creator includes within the archive the character mapping table for the source codeset used to encode extended header records. The receiver must then read the character mapping table and perform the appropriate translations to the destination codeset.

•

The archive creator translates the extended header records in the source codeset into a canonical form. The receiver must then perform the appropriate translations to the destination codeset.

The approach that incorporates the name of the source codeset poses the problem of codeset name registration, and makes the archive useless to pax archive decoders that do not recognize that codeset.

Because parts of an archive may be corrupted, the standard developers felt that including the character map of the source codeset was too fragile. The loss of this one key component could result in making the entire archive useless. (The difference between this and the global extended header decision was that the latter has a workaround-duplicating extended header records on unreliable media-but this would be too burdensome for large character set maps.)

Both of the above approaches also put an undue burden on the pax archive receiver to handle the cross-product of all source and destination codesets.

To simplify the translation from the source codeset to the canonical form and from the canonical form to the destination codeset, the standard developers decided that the internal representation should be a stateless encoding. A stateless encoding is one where each codepoint has the same meaning, without regard to the decoder being in a specific state. An example of a stateful encoding would be the Japanese Shift-JIS; an example of a stateless encoding would be the ISO/IEC 646:1991 standard (equivalent to 7-bit ASCII).

For these reasons, the standard developers decided to adopt a canonical format for the representation of file information strings. The obvious, well-endorsed candidate is the ISO/IEC 10646-1:2000 standard (based in part on Unicode), which can be used to represent the characters of virtually all standardized character sets. The standard developers initially agreed upon using UCS2 (16-bit Unicode) as the internal representation. This repertoire of characters provides a sufficiently rich set to represent all commonly-used codesets.

However, the standard developers found that the 16-bit Unicode representation had some problems. It forced the issue of standardizing byte ordering. The 2-byte length of each character made the extended header records twice as long for the case of strings coded entirely from historical 7-bit ASCII. For these reasons, the standard developers chose the UTF-8 defined in the ISO/IEC 10646-1:2000 standard. This multi-byte representation encodes UCS2 or UCS4 characters reliably and deterministically, eliminating the need for a canonical byte ordering. In addition, NUL octets and other characters possibly confusing to POSIX file systems do not appear, except to represent themselves. It was realized that certain national codesets take up more space after the encoding, due to their placement within the UCS range; it was felt that the usefulness of the encoding of the names outweighs the disadvantage of size increase for file, user, and group names.

The encoding of UTF-8 is as follows:

where each 'x' represents a bit value from the character being translated.

ustar Interchange Format¶

The description of the ustar format reflects numerous enhancements over pre-1988 versions of the historical tar utility. The goal of these changes was not only to provide the functional enhancements desired, but also to retain compatibility between new and old versions. This compatibility has been retained. Archives written using the old archive format are compatible with the new format.

Implementors should be aware that the previous file format did not include a mechanism to archive directory type files. For this reason, the convention of using a filename ending with slash was adopted to specify a directory on the archive.

The total size of the name and prefix fields have been set to meet the minimum requirements for {PATH_MAX} If a pathname will fit within the name field, it is recommended that the pathname be stored there without the use of the prefix field. Although the name field is known to be too small to contain {PATH_MAX} characters, the value was not changed in this version of the archive file format to retain backwards-compatibility, and instead the prefix was introduced. Also, because of the earlier version of the format, there is no way to remove the restriction on the linkname field being limited in size to just that of the name field.

The size field is required to be meaningful in all implementation extensions, although it could be zero. This is required so that the data blocks can always be properly counted.

It is suggested that if device special files need to be represented that cannot be represented in the standard format, that one of the extension types (A-Z) be used, and that the additional information for the special file be represented as data and be reflected in the size field.

Attempting to restore a special file type, where it is converted to ordinary data and conflicts with an existing filename, need not be specially detected by the utility. If run as an ordinary user, pax should not be able to overwrite the entries in, for example, /dev in any case (whether the file is converted to another type or not). If run as a privileged user, it should be able to do so, and it would be considered a bug if it did not. The same is true of ordinary data files and similarly named special files; it is impossible to anticipate the needs of the user (who could really intend to overwrite the file), so the behavior should be predictable (and thus regular) and rely on the protection system as required.

The value 7 in the typeflag field is intended to define how contiguous files can be stored in a ustar archive. IEEE Std 1003.1-2001 does not require the contiguous file extension, but does define a standard way of archiving such files so that all conforming systems can interpret these file types in a meaningful and consistent manner. On a system that does not support extended file types, the pax utility should do the best it can with the file and go on to the next.

The file protection modes are those conventionally used by the ls(1) utility. This is extended beyond the usage in the ISO POSIX-2 standard to support the "shared text" or "sticky" bit. It is intended that the conformance document should not document anything beyond the existence of and support of such a mode. Further extensions are expected to these bits, particularly with overloading the set-user-ID and set-group-ID flags.

cpio Interchange Format¶

The reference to appropriate privilege in the cpio format refers to an error on standard output; the ustar format does not make comparable statements.

The model for this format was the historical System V cpio -c data interchange format. This model documents the portable version of the cpio format and not the binary version. It has the flexibility to transfer data of any type described within IEEE Std 1003.1-2001, yet is extensible to transfer data types specific to extensions beyond IEEE Std 1003.1-2001 (for example, contiguous files). Because it describes existing practice, there is no question of maintaining upwards-compatibility.

cpio Header¶

There has been some concern that the size of the c_ino field of the header is too small to handle those systems that have very large inode numbers. However, the c_ino field in the header is used strictly as a hard-link resolution mechanism for archives. It is not necessarily the same value as the inode number of the file in the location from which that file is extracted.

The name c_magic is based on historical usage.

cpio Filename¶

For most historical implementations of the cpio utility, {PATH_MAX} octets can be used to describe the pathname without the addition of any other header fields (the NUL character would be included in this count). {PATH_MAX} is the minimum value for pathname size, documented as 256 bytes. However, an implementation may use c_namesize to determine the exact length of the pathname. With the current description of the <cpio.h> header, this pathname size can be as large as a number that is described in six octal digits.

Two values are documented under the c_mode field values to provide for extensibility for known file types:

0110 000

Reserved for contiguous files. The implementation may treat the rest of the information for this archive like a regular file. If this file type is undefined, the implementation may create the file as a regular file.

This provides for extensibility of the cpio format while allowing for the ability to read old archives. Files of an unknown type may be read as "regular files" on some implementations. On a system that does not support extended file types, the pax utility should do the best it can with the file and go on to the next.

Issue 5¶

A note is added to the APPLICATION USAGE indicating that the cpio and tar formats can only support files up to 8 gigabytes in size.

Issue 6¶

The pax utility is aligned with the IEEE P1003.2b draft standard:

•

Support has been added for symbolic links in the options and interchange formats.

•

A new format has been devised, based on extensions to ustar.

•

References to the "extended" tar and cpio formats derived from the POSIX.1-1990 standard have been changed to remove the "extended" adjective because this could cause confusion with the extended tar header added in this revision. (All references to tar are actually to ustar.)

The TZ entry is added to the ENVIRONMENT VARIABLES section.

IEEE PASC Interpretation 1003.2 #168 is applied, clarifying that mkdir(2) and mkfifo(3) calls can ignore an [EEXIST] error when extracting an archive.

IEEE PASC Interpretation 1003.2 #180 is applied, clarifying how extracted files are created when in read mode.

IEEE PASC Interpretation 1003.2 #181 is applied, clarifying the description of the -t option.

IEEE PASC Interpretation 1003.2 #195 is applied.

IEEE PASC Interpretation 1003.2 #206 is applied, clarifying the handling of links for the -H, -L, and -l options.

IEEE Std 1003.1-2001/Cor 1-2002, item XCU/TC1/D6/35 is applied, adding the process ID of the pax process into certain fields. This change provides a method for the implementation to ensure that different instances of pax extracting a file named /a/b/foo will not collide when processing the extended header information associated with foo.

IEEE Std 1003.1-2001/Cor 1-2002, item XCU/TC1/D6/36 is applied, changing -x B to -x pax in the OPTIONS section.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/20 is applied, updating the SYNOPSIS to be consistent with the normative text.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/21 is applied, updating the DESCRIPTION to describe the behavior when files to be linked are symbolic links and the system is not capable of making hard links to symbolic links.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/22 is applied, updating the OPTIONS section to describe the behavior for how multiple options are to be handled.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/23 is applied, updating the write option within the OPTIONS section.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/24 is applied, adding a paragraph into the OPTIONS section that states that specifying more than one of the mutually-exclusive options (-H and -L) is not considered an error and that the last option specified will determine the behavior of the utility.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/25 is applied, removing the ctime paragraph within the EXTENDED DESCRIPTION. There is a contradiction in the definition of the ctime keyword for the pax extended header, in that the st_ctime member of the stat structure does not refer to a file creation time. No field in the standard stat structure from <sys/stat.h> includes a file creation time.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/26 is applied, making it clear that typeflag 1 RB ( ustar Interchange Format) applies not only to files that are hard-linked, but also to files that are aliased via symlinks.

IEEE Std 1003.1-2001/Cor 2-2004, item XCU/TC2/D6/27 is applied, clarifying the cpio c_nlink field.

End of quoted text from the POSIX.1-2001 standard.