Opera File Formats

Introduction

This document describes the new binary file formats introduced with Opera version 5 for the various files used in the cache and cookie management.

The new generic format that these files are based on is structured as a sequence of tagged records with a given length. Each record may contain a number of different data types such as strings and integers, as well as arbitrary binary data, such as new records in this format.

The generic format is NOT backwards compatible with Opera 3.x and earlier, but is intended to be reasonably backwards compatible for version 4.0 and later. This means that if new fields are added by a future version, older versions will still be able to read the information that they understand, while ignoring the fields they do not understand.

There are some limits, but they are mostly concerned with the number of significant bits in the integers that are used to indicate record lengths. The formats are presently used to store the disk cache index file (dcache4.url), the visited links file (vlink4.dat), the download rescue file (download.dat) and the cookie file (cookies4.dat). The formats are described in the following sequence:

• The generic binary format used in the new formats.
• The special constants and data organizations used in the cache, visited links and download file formats.
• The special constants and data organizations used in the cookie format.

The formats for the windows history, news files and global history files are the same as the ones used by Opera v3.x and are outside the scope of this document.

The Generic Binary Format

Data types

Integers used in the format are unsigned, and stored in big-endian/network style (Most Significant Byte first). Integers stored inside the records are also stored in the big-endian format, but may be signed, and may be truncated.

The following datatypes are used in this document:

Data records

The general record format has this form:

struct record
{
// application specific tag to identify content type
tag_id_type tag_id;
// length of payload
payload_length_type length;
// Payload/content of the record
bytepayload[length];
};

NOTE: The number of bytes in tag and length may change, see below.

The fields of each record have the following meanings:

tag_id

The identifier of the record. This value is application specific, and can be used to indicate the meaning of the payload content.

The actual content type of the record depends on the definitions used for the actual file or super-record.

Tag_id values in which the MSB (Most Significant Bit) is set to 1, are reserved for records with implicit no length. The tag_id field is NOT followed by a length field, nor a payload buffer. Such records are used as Boolean flags: True if present, False if not present.

In the binary storage of a file this means that the MSB of the internal storage integer must be stored as the MSB of the first byte in the tag field. This places a limit on how many tags can be used for a given tag_id integer length. When a file is read into a program, the program must take care to move the MSB of the binary stored tag to a common (internal) bit position, such as the MSB of the program's own unsigned integers.

bytes	Max id available (excluding MSB)
1	0x7f
2	0x7fff
3	0x7fffff
4	0x7fffffff

While it is technically possible to use the same tag id (without the MSB) for a normal record and a flag record (such as 0x0001 [16 bit tag] for the payload record and 0x8001 for the flag record), this is not encouraged.

length

This field is the number of bytes in the payload that immediately follow the field. It may be zero.

payload

The payload is a sequence of bytes of the length indicated by the length field.

The meaning of the contents is indicated by the definition for the given record or file structure. Examples of organization may be an array of records, unsigned integers, signed integers, or characters.

It is recommended that only records of the types described here are used if the type of the data varies, as variable (un-tagged) type formats tend to be inflexible and difficult to maintain across versions, especially when compatibility with older versions is desired.

Single item integers (signed or unsigned) may be truncated (zero bytes removed), but arrays of integers must always use a fixed number of bytes to represent values and derive the number of items from the payload length. If the number of bytes needed to represent the values changes in a future version a new tag should be used.

File Format

These elements are not stored as records but directly in binary:

uint32 file_version_number;
uint32 app_version_number;
// number of bytes in the id tag, presently 1
uint16 idtag_length;   
// number of bytes in the length part of a record, presently 2
uint16 length_length;  
// array of records, number determined by length of file
struct record items[]; 

The present version number of the file format (file_version_number) is 0x00001000, where the lower 12 bits (bitmask 0x00000fff) represent the minor version number, the rest is the major version number. Changes in the minor version must not be used if the file format is changed in such a manner that older versions of the software cannot read the file successfully. If the major version number is newer (or older) than the application can read, it must not read the file.

The integer sizes are absolute for a given major version, and the integer size for the file version number is fixed in any version.

The "app_version_number" is the version number of the application and is independent of the file_version_number. It may be used by the application to determine necessary actions needed to provide forward or backward compatibility that is outside the scope of the file formats. The interpretation of the application version number is application dependent.

The "idtag_length" and "length_length" fields gives the number of bytes used in the records for the idtags, as defined by the tag_id_type, and the payload length fields, as defined by payload_length_type, respectively.

Specifically, the values of these fields define tag_id_type and payload_length_type as the following integer types:

The application's internal representation of these types is not defined, but uint32 is recommended. How an application should handle "idtag_length" or "length_length" values larger than 4, or values larger than its internal unsigned integer size, is not defined, but the application should implement the rules specified in the forward compatibility guide for such situations.

Present versions of Opera 4.x uses idtag_length=1 (uint8) and length_length=2 (uint16).

After the header, only records follow. The organization of the records and their interpretation is application specific.

Forward Compatibility

An older version of an application using this file format that is NOT able to use long integers should, regardless of this, try to process the file, but should bypass the record if the tag of the record's numerical value exceeds the version's own integer range, i.e. the integer overflows. However, if the the length of the record exceeds the application's limits on integers or buffer capabilities, it must not continue to process the file.

All applications must ignore tag values that they do not understand.

The Cache File Formats

This section details the record tags and formats used for the visited link file (vlink4.dat), the disk cache index file (dcache4.url) and the download rescue file (download.dat). The present app_version_number of these files is 0x00020000 (major version 2, minor 0).

These files use records (of different tag values) which contain a sequence of records with tags from the same set of tag ids. The different files use these tags for their records:

Each file consists of records of ONLY this type, with the exception of the Disk cache index file, which also contains a single record with the id 0x40, which contains a 5 character string used to find the next free cache file number (oprXXXXX).

Each record is again a sequence of records with the same binary representation format as the records in the file.

Common Elements Between All Files

These elements are used by all of the cache related files. In the case of the visited links, these are the only fields presently used.

NOTE: "(0x0001 | MSB_VALUE)" means that the most significant bit in the local unsigned integer is to be set. If 32 bit values are used, that means the tag's value is 0x80000001.

Content tags of relative link (tag 0x0022) records

Fields Used by Disk Cache and Download Rescue Files

Fields used only by the download rescuefile

Fields used in the HTTP protocol specific record

All methods are by default GET, at present it is not possible to cache POST requests.

Cookie File format

This section describes the record tags and formats used for the storage of cookies (cookies4.dat). The present app_version_number of this file type is 0x00002000 (major version 2, minor 0).

The cookie file is organized as a tree of domain name components, each component then holds a tree of path components and each path component may contain a number of cookies.

NOTE: The components are a sequence of records, teminated with a flag record, not a single record.

Structure

Domain components

The domain components are used to organize the cookies for each server and domain for which cookies or cookie filtering capabilities are defined.

A domain component is started with a domain record, which holds the domain name and some flags for that particular domain. It is then followed by a path component holding the cookies and subdirectory path components (and cookies), followed with a path component terminator and any number of subdomain components before it is terminated by a domain-end flag record.

E.g: cookies for the domain www.opera.com will be stored in this manner:

["com" record]
  ["opera" record]
["www" record
  [cookies]
  [Path components]
  [Path component terminator]
  [other domains]
[end of domain flag ("www")]
  [end of domain flag ("opera")]
[end of domain flag ("com")]

All names of domain components are non-dotted, except IP addresses, which can only be stored with the complete IP address as a Quad dotted string, e.g. "10.11.12.13", are stored at the top level, and cannot contain any subdomains.

A Domain Record uses the tag "0x01" and contains a sequence of these fields:

This record can be followed by zero or more path components defining toplevel paths on servers in the domain and always terminated by a path component terminator record. Then zero or more domain components may follow.

A domain component is terminated by a (0x0004 | MSB_VALUE) flag record.

Path Components

The path components organize the cookies defined for a given directory in a given domain, as well any subdirectories of this directory that have cookies defined.

Except for the path component starting immediately after the domain component record, each path component always starts with a path record, and is then followed by any number of cookie records and subdirectory path components.

The path record uses the record id "0x0002" and the record has this field record:

The path component terminator is the (0x0005 | MSB_VALUE) flag record.

Cookie Records

The cookie entries are stored in records of type "0x0003" and have the following field records:

Copyright Opera Software ASA. All rights reserved. | Support | Contact | Privacy | Jobs