Data Description Header

1 Overview

The Data Description Header (DDH) is a standard way for files of all types to declare their format and (if appropriate) their Unicode encoding. It is designed to replace a variety of heuristics such as file extensions, 4-byte magic numbers and file system attributes. Each of these heuristic solutions has a weakness that prevents it from being universally used. The DDH is embedded in the same data stream as the data content so that it will not get separated from it under normal circumstances.

The DDH is designed to be as unobtrusive as possible, and yet extensible to handle more complex problems. Here are a few examples of gradually increasing complexity.

Declaring a stream to be a Java program:

<?application/java?>

Declaraing a stream to be a Python 2.3 program:

<?application/python version="2.3"?>

Declarating a a stream to be a Shift-JIS encoded Perl 6 program:

<?application/perl version="6" encoding="Shift-JIS"?>

Declarating a stream to be an RTF document, including documentation about the RTF specifrication:

<?text/rtf version="1.5" encoding="ASCII"
	DocURI="http://www.biblioscape.com/rtf15_spec.htm"?>

Declaring a stream to be a (binary) Zip file.

<?application/zip version="1.0" encoding="ASCII" dataEncoding="binary" DocURI="http://www.pkware.com/products/enterprise/white_papers/appnote.html"?>

Adding some metadata to the Zip file:

<?application/zip version="1.0" encoding="ASCII" dataEncoding="binary" DocURI="http://www.pkware.com/products/enterprise/white_papers/appnote.html"?>
	<xml:meta  
      xmlns:dc="http://purl.org/dc/elements/1.0/"
       xmlns:zip="http://www.pkware.com/zip"><dc:Identifier id="uid">261F1B8C-6C81-11D3-8BFC-0050E4009B3F</dc:Identifier>
             <dc:Title>Open eBook Publication Structure 1.0.1</dc:Title>
				 <dc:Creator role="author">Paul Prescod</dc:Creator><zip:manifest> ...</zip:manifest>
   

Embedding a header in a pre-existing file format without explicit support for this DDH

#!/usr/bin/python
	# <?application/python?>

2 Rationale

There are three related problems that this specification is designed to solve.

3 Normative Definitions

stream

A stream is a sequence of bytes starting with a header defined by this specification.

stream

[1]

document

::=

(header|extendedHeader) separator body

header

A header is a stream of bytes in some Unicode encoding (including historical national encodings such as ASCII, Shift-JIS, etc.). The algorithm for auto-detecting the encoding is the same as that for XML. Just as with XML, the string "<?" is a reliable key for lookup in an encodings table.

A header is a stream of bytes in some Unicode encoding (including historical national encodings such as ASCII, Shift-JIS, etc.). The algorithm for auto-detecting the encoding is the same as that for XML. These production (and all referenced by it) refer to the post-decoding character sequence.

Headers

[2]

header

::=

declaration metadata?

Type Declaration

The type declaration states the media-type, Unicode encoding of the header, data encoding of the body (if different).

Type Declaration

[3]

typeDeclaration

::=

'<?' typeID? versionInfo? encodingDecl? docURI? dataEncodingDecl?' ?>'

Datatype Declaration

The datatype declaration declares the overall format of the data stream. For example, it could be a Python module or a Word document. It can be a straightforward media-type (as defined in HTTP and MIME or an indentifying URI.

Type Identifier Declaration

[4]

Type Identifier

::=

mediaType | TypeURI

Type Description URI

The TypeURI is a type identifier in URI rather than MIME syntax.

Ideally, it can be dereferenced to return information that could be both human and machine readable. Two media types with different TypeURIs are presumed to be different for the purposes of this specification (just as if they were declared with two distinct MIME types).

Note:

The editor hopes that there will arise an RDF schema that can express proper subtype/supertype relationships between formats so that applications can recognize when a document is "similar enough" to what they understand to be processable.

Type Documentation URI

The DocURI is a pointer to human or machine readable documentation about the data format. It is distinguished from the TypeURI in thatit is not considered an identifier. You could point to one URI for information about the ZIP file format and I could point to another.

Version Information

VersionInfo is any string that meets the XML production of the same name. Its meaning is assigned by the description of the MIME type and not constrained by this specification.

Encoding Declaration

The Encoding declaration is as defined in XML. Just as in XML, it is optional if the data is UTF-8 or UTF-16 or in the rare case that the encoding can be reliably inferred from the stream's context.

Data Encoding Declaration

The DataEncodingDecl is a pseudo-attribute named "dataEncoding". It defines the Unicode encoding not for the header but for the Body. The value "binary" is used to indicate that no Unicode decoding should be attempted for the Body. If the DataEncodingDecl is omitted, it defaults to the same encoding as the header (which may have been inferred to be UTF-8 or UTF-16).

Data Encoding Declaration

[5]

dataEncodingDecl

::=

'dataEncodingDecl' S? = S? encoding | "binary"

Metadata

The metadata part of the header (if provided) should be a well-formed XML document (with or without an XML declaration). The root element must have the element type "xml:meta". Each sub-element or root-level attribute must be in a namespace. Processors (including editors) should ignore elements or attributes in namespaces they are not programmed to recognize. They SHOULD not remove unknown elements or attributes in namespaces they do not understand. If the XML declaration has an encoding declaration, it MUST NOT be different than that for the header as a whole.

Separator

If the Body is in a different encoding than the header (typically binary) then the separator must be the character sequence FF, SUB, EOT (aka "^L^Z^D" aka "FORM FEED", "SUBSTITUTE", "END OF TRANSMISSION") which should serve to visually separate the text from the binary data in the terminal programs of most computers.

Otherwise, the separator is considered to be empty and the body begins with the first character in the stream that could not be interepreted as part of the XML document (i.e. is outside of markup and is neither whitespace nor "<"). This necessarily precludes the body from starting with "<" or whitespace. New formats MUST be defined accordingly.

Note:

This requirement is not burdensome. At worst, a format could define its own separator outside of the scope of this specification and discard the separator before processing the body.

Separator

[6]

separator

::=

(#0C #1A #03)?

4 Backwards Compatibility Issues

This specification defines a mechanism called the "extended header". It is designed to support pre-existing uses for the first lines of files. This specification does not change the definition of any pre-existing media types. They should be interpreted as per their various specifications. For example, most Unix systems will not support UTF-16 shell scripts even though this specification might allow such a declaration.

The specification does, however, allow the addition of metadata to those media types for software applications that understand this specification.

It is anticipated that specifications for new formats will make normative references to this one so that this mechanism can replace the various ad hoc mechanisms for self-description and inline metadata. The extended header is merely an interim solution for older formats.

#!/usr/bin/python2.3	
# <?application/python version="2.3"?>
import x	
import y	
print "z"

/*<?application/java?> */
class { ...