AV2 Codec ISO Media File Format Binding

1. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes.

Examples in this specification are introduced with the words "for example" or are set apart from the normative text with class="example", like this:

Informative notes begin with the word "Note" and are set apart from the normative text with class="note", like this:

NOTE: This is an example of an informative note.

↑ Back to Table of Contents

2. Bitstream features overview

An AV2 bitstream is composed of a sequence of OBUs (Open Bitstream Units), grouped into Temporal Units. A Temporal Unit consists of a Temporal Delimiter OBU and all following OBUs up to, but not including, the next Temporal Delimiter OBU. Each Temporal Unit is associated with a single presentation time.

Every OBU begins with a 1- or 2-byte header that carries the obu_type and the temporal layer identifier obu_tlayer_id. When the OBU header extension is present (i.e. when obu_header_extension_flag is equal to 1 in that header), two additional layer identifiers are carried: the embedded layer identifier obu_mlayer_id and the extended layer identifier obu_xlayer_id. Depending on its type, an OBU can carry configuration information (e.g., a Sequence Header OBU or Layer Configuration Record OBU), metadata, or coded video data.

A Temporal Unit MAY contain multiple coded pictures but only one is output. AV2 defines a number of coded-frame OBU types, including Closed Loop Key OBUs, Open Loop Key OBUs, regular and leading tile-group OBUs, Switch OBUs, OBU_RAS_FRAME, Bridge Frame OBUs, and TIP (Temporally Interpolated Prediction) frames.

Frames that can be decoded without reference to other frames are Key Frames (carried in a Closed Loop Key OBU or an Open Loop Key OBU) and Intra frames. All other coded frames - including Inter Frames, Switch Frames, RAS frames, Bridge frames, TIP frames, and frames with show_existing_frame equal to 1 - have coding dependencies on other frames in the same or previous Temporal Units.

A Closed Random Access is the random access process that applies to an extended layer whose first coded frame is a Key Frame carried in a Closed Loop Key OBU. It starts a new coded video sequence for the extended layer; the layer’s reference frame buffers are invalidated and every subsequent frame of that layer can be decoded. This is the primary random-access mechanism.

An Open Random Access is the random access process that applies to an extended layer whose first coded frame is a Key Frame carried in an Open Loop Key OBU. During sequential decoding, an Open Random Access point does not start a new coded video sequence; however, when a decoder initiates decoding at that point, the decoding process is treated as if it were the start of a new coded video sequence for the extended layer. Leading frames that follow an Open Random Access point MAY reference frames that precede it, and are discarded when decoding begins at the Open Random Access point.

In addition to Key Frames, AV2 provides two further mechanisms that support random access or stream switching: a RAS frame (OBU_RAS_FRAME) is an inter-predicted frame that uses long-term reference frames and enables random access or switching without inserting a full key frame; a Switch Frame (OBU_SWITCH) is an inter-predicted frame that enables switching between representations at a given point, typically in adaptive streaming scenarios.

AV2 supports three scalability dimensions: temporal layers (identified by obu_tlayer_id), embedded layers (identified by obu_mlayer_id), and extended layers (identified by obu_xlayer_id). The reserved obu_xlayer_id value GLOBAL_XLAYER_ID (equal to 31, as defined in [AV2]) denotes global scope and is used by OBUs that do not belong to any specific extended layer (such as the Temporal Delimiter OBU). A bitstream that uses a single distinct non-global value of obu_xlayer_id is a Singlestream bitstream; a bitstream that uses two or more distinct non-global values is a Multistream bitstream.

AV2 defines two distinct Metadata OBU types for carrying metadata: OBU_METADATA_SHORT, using the metadata short OBU syntax, and OBU_METADATA_GROUP, using the metadata group OBU syntax. A separate OBU type carries film grain synthesis parameters; its presence is indicated by the film_grain_params_present flag in the Sequence Header OBU.

The Sequence Header OBU also carries the bitstream’s profile, level, and - when applicable - tier, as defined in Annex A of [AV2].

The Sequence Header OBU carries a monotonic_output_order_flag that defines the output mode of the associated coded video sequence.

When monotonic_output_order_flag is equal to 1, the output order of coded output frame equals their decoding order within the coded video sequence. Within a sample, the composition time (CTS) of the associated output frame is equal to the sample’s decoding time (DTS); the sample MAY additionally contain hidden frames.

When monotonic_output_order_flag is equal to 0, the output order of coded output frames MAY differ from their decoding order. The composition time of each sample is then derived from a ctts box that signals the offset between the sample’s decoding time (DTS) and the composition time of its associated output frame. As in the monotonic case, a sample MAY additionally contain hidden frames.

NOTE: Non-monotonic output mode is new in AV2; AV1 supports only the monotonic mode.

↑ Back to Table of Contents

3. Basic Encapsulation Scheme

This section describes the basic data structures used to signal encapsulation of AV2 bitstreams in [ISOBMFF] containers.

There are two locations where OBUs can be found in a file conformant to this specification:

• In out-of-band mode, some High-Level Syntax OBUs, such as the Sequence Header OBU, are carried in the AV2CodecConfigurationBox as described in § 3.2 AV2 Sample Entry and Configuration. This corresponds to "availability through external means" as described in the [AV2] specification. These OBUs apply to all samples of the corresponding AV2SampleEntry.

• The other OBUs, and all OBUs in in-band mode, are stored in samples. This corresponds to "presence in the bitstream" as described in the [AV2] specification.

An AV2 sample is the union of a sample inclusive of the OBUs it carries, the OBUs carried in the AV2CodecConfigurationBox of the corresponding AV2SampleEntry, and an implicit Temporal Delimiter OBU.

NOTE: This specification refers to samples as defined in [ISOBMFF], which are not to be mistaken with samples or sample values as defined in the [AV2] specification.

3.1. General Requirements & Brands

A file conformant to this specification satisfies the following:

• It SHALL conform to the normative requirements of [ISOBMFF]
• It SHALL have the av02 brand among the compatible brands array of the FileTypeBox
• It SHALL contain at least one track using an AV2SampleEntry, possibly transformed by encryption as specified in § 5 Common Encryption
• It SHOULD indicate a structural ISOBMFF brand among the compatible brands array of the FileTypeBox, such as iso6
• It SHALL contain at least one AV2 sample
• Each AV2 sample it contains SHALL form exactly one [AV2] Temporal Unit if reconstructed as described in § 3.3.6 AV2 sample reconstruction

3.2. AV2 Sample Entry and Configuration

3.2.1. AV2 Sample Entry

The AV2SampleEntry extends VisualSampleEntry and signals a track carrying AV2 samples.

class AV2SampleEntry extends VisualSampleEntry('av02') {
  AV2CodecConfigurationBox config;
}

• SHOULD equal the corresponding effective value of the AV2 bitstream carried by the track if the latter is anything but 2 (Unspecified);
• SHALL take precedence over the corresponding effective value of the AV2 bitstream carried by the track.

NOTE: An AV2SampleEntry may contain additional sub-boxes permitted by [ISOBMFF] for a VisualSampleEntry, such as colr with a colour_type value other than nclx, pasp, clli, mdcv or btrt. They may be located before and/or after the AV2CodecConfigurationBox.

3.2.2. AV2 Codec Configuration Box

The AV2CodecConfigurationBox carries the decoder initialisation data required to decode the AV2 samples of a track.

aligned(8) class AV2CodecConfigurationBox extends FullBox('av2C', 0, av2c_flags)
{
  unsigned int(5) profile_idc;
  unsigned int(5) level_idx;
  unsigned int(1) tier;
  unsigned int(1) initial_presentation_delay_present;
  if (initial_presentation_delay_present) {
    unsigned int(4) initial_presentation_delay_minus_one;
  } else {
    unsigned int(4) reserved1 = 0;
  }
  if ((av2c_flags & multistream_present) != 0) {
    unsigned int(16) max_canvas_width_minus_1;
    unsigned int(16) max_canvas_height_minus_1;
    unsigned int(3) max_num_substreams_minus2;
    unsigned int(5) max_xlayer_cnt_minus1;
    unsigned int(3) max_mlayer_cnt_minus1;
    unsigned int(1) atlas_present;
    unsigned int(1) alpha_atlas_present;
    unsigned int(1) lcr_present;
    unsigned int(2) reserved2 = 0;
  }
  unsigned int(8) config_obus_count;
  for (i = 0; i < config_obus_count; ++i) {
    unsigned int(8) config_obu[];
  }
}

av2c_flags is a 24-bit field. The following flag is defined:

• multistream_present: flag value 0x000001. When set, it indicates that the track carries a Multistream bitstream and that the multistream fields are present. It SHALL be set if and only if the track contains a Multi Stream Decoder Operation OBU (OBU_MSDO).

profile_idc, level_idx and tier are the profile, level and tier required to decode the track. For a track that is not a Multistream, they SHALL equal the maximum of seq_profile_idc, seq_level_idx and seq_tier over the Sequence Header OBUs applying to the AV2SampleEntry. For a Multistream track, they SHALL equal the maximum of multistream_profile_idc, multistream_level_idx and multistream_tier over the Multi Stream Decoder Operation OBUs applying to the AV2SampleEntry.

initial_presentation_delay_present and initial_presentation_delay_minus_one signal, when present, the number of frames minus one that the decoder should wait after the first sample’s decode before starting output.

max_canvas_width_minus_1 and max_canvas_height_minus_1 are the maximum canvas dimensions, minus one, of any composition of the track. max_num_substreams_minus2, max_xlayer_cnt_minus1 and max_mlayer_cnt_minus1 are the maximum number of sub-streams minus two, extended layers minus one and embedded layers minus one, over the whole track. atlas_present, alpha_atlas_present and lcr_present indicate that the track contains, respectively, OBU_ATLAS_SEGMENT OBUs, alpha atlas segments and Layer Configuration Record OBUs.

NOTE: Bit depth and chroma format are not repeated in the AV2CodecConfigurationBox: they are described by the mandatory pixi box of the AV2SampleEntry, as specified in § 3.3.5 Summary information in the pixi box.

config_obus_count is the number of config_obu. A value of 0 selects the in-band carriage mode described in § 3.2.3 Carriage modes.

config_obu is a byte array carrying one OBU and its length as described in § 3.2.2.1 config_obu.

3.2.2.1. config_obu

• the OBUs within an AV2CodecConfigurationBox SHALL NOT contain OBUs of the types forbidden there as stated in Figure 3.1.

Editor’s note: for a Multistream bitstream, the AV2CodecConfigurationBox contains a Sequence Header OBU for each extended layer. The impact on multi-layer use cases — including how a reader determines the active decoder configuration per extended layer from multiple Sequence Header OBUs — needs investigation. Also how profile / level is signalled to decoder API for layers. etc.

3.2.3. Carriage modes

An AV2SampleEntry uses one of two carriage modes for the High-Level Syntax OBUs that it applies to:

• Out-of-band mode: the High-Level Syntax OBUs are carried in the AV2CodecConfigurationBox (config_obus_count greater than 0).

• In-band mode: the High-Level Syntax OBUs, including the Multi Stream Decoder Operation and Atlas Segment OBUs, are carried in the samples (config_obus_count equal to 0).

3.2.4. Configuration changes

In out-of-band mode, a change of configuration SHOULD be signalled by a new AV2SampleEntry, selected through sample_description_index in the SampleToChunkBox (stsc) or the TrackFragmentHeaderBox (tfhd).

In in-band mode, a change of configuration is signalled by new High-Level Syntax OBUs in the next sync AV2 sample. In in-band mode, the configuration SHALL NOT change at an AV2 sample that is not a sync AV2 sample. The summary fields of the AV2CodecConfigurationBox describe the maximum capability required for the complete track.

3.3. AV2 Sample Format

3.3.1. General

The AV2 sample boundary delimits the temporal unit.

3.3.2. OBUs in samples

• the OBUs within a sample SHALL NOT contain OBUs of the types forbidden there as stated in Figure 3.1.

Editor’s note: the Content Interpretation OBU (OBU_CONTENT_INTERPRETATION) carries picture-level interpretation metadata (for example, colour description) that overlaps ISOBMFF container-level signalling such as the ColourInformationBox. Whether to carry it in the track or replace it entirely with container-level signalling is to be resolved in this version.

3.3.3. Sync sample

An AV2 sample is a sync AV2 sample if, for every coded extended layer unit present in the AV2 sample, the first coded frame OBU of that coded extended layer unit has obu_type equal to OBU_CLOSED_LOOP_KEY. Such an AV2 sample is a Closed Random Access point for the carried bitstream and corresponds to a SAP of type 1 or 2 as defined in Annex I of [ISOBMFF].

NOTE: In ISOBMFF terms, sync samples are signalled using the SyncSampleBox (stss) in unfragmented tracks, or by setting sample_is_non_sync_sample equal to 0 in fragmented tracks. When every sample in a track is a sync sample, the SyncSampleBox is omitted in unfragmented tracks (per [ISOBMFF]), and in fragmented tracks this is expressed through the default sample flags of the MovieExtendsBox (trex) or TrackFragmentHeaderBox (tfhd) with no per-sample override.

3.3.4. Layers

The following applies to each coded extended layer unit in an AV2 sample associated with a pixi box:

• The channels in the AV2 sample are considered to be ordered as if the embedded layers were concatenated in increasing mlayer_id order.

  An embedded layer with mlayer_id 0 and CHROMA_FORMAT_420 contains the three first channels. Another embedded layer with mlayer_id 1 and CHROMA_FORMAT_400 contains the fourth channel.
• If the pixi box describes the Nth channel as used and specified (channel_idc values other than 0 and 1), the Nth channel in the AV2 sample SHALL be present.
• If the pixi box describes the Nth channel to be a color channel (channel_idc values such as 2, 3, 4, and 7), the Nth channel in the AV2 sample SHALL be a color channel: if signalled, the lcr_layer_type field of its associated Layer Configuration Record OBU SHALL be set to 0 (TEXTURE_LAYER).
• If the pixi box describes the Nth channel to be an alpha channel (channel_idc 5), the Nth channel in the AV2 sample SHALL be an alpha channel: if signalled, the lcr_layer_type field of its associated Layer Configuration Record OBU SHALL be set to 1 (AUX_LAYER) and lcr_auxiliary_type to 0 (ALPHA_AUX).
• If the pixi box describes the Nth channel to be a depth channel (channel_idc 6), the Nth channel in the AV2 sample SHALL be a depth channel: if signalled, the lcr_layer_type field of its associated Layer Configuration Record OBU SHALL be set to 1 (AUX_LAYER) and lcr_auxiliary_type to 1 (DEPTH_AUX).
• If the pixi box describes the Nth channel as used and specified (channel_idc values other than 0 and 1) with subsampling_flag set to 1, the Nth channel in the AV2 sample SHALL have the value of the chroma_format_idc field of its associated Sequence Header OBU correspond to the same meaning as the value of the pixi box’s Nth channel’s subsampling_type value.
• If the pixi box describes the Nth channel as used and specified (channel_idc values other than 0 and 1) with subsampling_flag set to 1 and the Nth channel in the AV2 sample is associated with a Content Interpretation OBU with ci_chroma_sample_position_present_flag set to 1, the value of the ci_chroma_sample_position_top field of that Content Interpretation OBU SHALL correspond to the same meaning as the value of the pixi box’s Nth channel’s subsampling_location value.

3.3.5. Summary information in the pixi box

The AV2CodecConfigurationBox carries profile, level and tier but no bit depth or chroma format. Readers obtain them from the pixi box of the AV2SampleEntry, in addition to the channel rules of § 3.3.4 Layers:

Editor’s note: pixi semantics follow the draft FDIS of [ISOBMFF] 9th edition (Tables 8 and 9) and need re-checking against the published text. Open: the component_format value for unsigned integer (ISO/IEC 23001-17); 4:2:2 subsampling_location mapping to AV2 chroma sample positions 0 and 1; channel_idc for channels with no defined meaning (segmentation, gain map).

3.3.6. AV2 sample reconstruction

A byte sequence containing a conformant [AV2] Temporal Unit using the Length delimited bitstream format specified in Annex B of [AV2] MAY be reconstructed from the OBUs of an AV2 sample by applying the following steps:

1. Write a Temporal Delimiter OBU, preceded by its leb128() length prefix.

2. Write the OBUs carried in the sample, in the same order as present in the sample, each preceded by its leb128() length prefix as present in the sample.

3. In out-of-band mode, insert each OBU carried in the AV2CodecConfigurationBox of the corresponding AV2SampleEntry, preceded by its leb128() length prefix as present in the AV2SampleEntry, between two leb128()-prefixed OBUs already present in the byte sequence, so that:

   • the relative order of the OBUs carried in an AV2CodecConfigurationBox SHALL be kept in the reconstructed byte sequence,

   • the reconstructed byte sequence SHALL conform to the ordering of OBUs in a temporal unit as specified in [AV2], and

   • there SHALL NOT be an earlier insertion location for that OBU that would satisfy the above constraints.

Note: AV2-ISOBMFF reader implementations do not have to reconstruct the contiguous byte sequence containing a conformant AV2 bitstream or Temporal Unit in memory upon decoding, but they must have all the data to do so: all OBUs, and enough information to deterministically reorder the OBUs if necessary.

↑ Back to Table of Contents

4. CMAF AV2 track format

TBD

↑ Back to Table of Contents

5. Common Encryption

TBD

↑ Back to Table of Contents

6. Codecs Parameter String

DASH and other applications require defined values for the Codecs parameter specified in [RFC6381] for ISO Media tracks. The codecs parameter string for the AOM AV2 codec is as follows:

<sample entry 4CC>.<profile>.<level><tier>.<bitDepth>.<monochrome>.<chromaSubsampling>.
<colorPrimaries>.<transferCharacteristics>.<matrixCoefficients>.<videoFullRangeFlag>

All fields following the sample entry 4CC are expressed as double digit decimals, unless indicated otherwise. Leading or trailing zeros cannot be omitted.

The chromaSubsampling parameter value, represented by a three-digit decimal:

1. SHALL have its first digit equal to the SubsamplingX variable as defined in [AV2] derived from the first Sequence Header OBU.
2. SHALL have its second digit equal to the SubsamplingY variable as defined in [AV2] derived from the first Sequence Header OBU.
3. If SubsamplingX is set to 1, a Content Interpretation OBU is present with the same obu_tlayer_id, obu_mlayer_id, obu_xlayer_id as the first Sequence Header OBU, and ci_chroma_sample_position_present_flag is set to 1 in that Content Interpretation OBU, then the third digit SHALL be equal to ci_chroma_sample_position_top from that Content Interpretation OBU, otherwise it SHALL be set to 0.

NOTE: A colr box with the nclx colour_type is mandated to be present in § 3.2.1 AV2 Sample Entry.

NOTE: The third digit values of the chromaSubsampling parameter differs from how they are defined in AV1-ISOBMFF. The values of the other fields of the Codecs parameter have the same meaning in this document and in AV1-ISOBMFF.

The videoFullRangeFlag is represented by 0 or 1.

For example, codecs="av02.0.04M.10.0.112.09.16.09.0" represents [AV2] Main_420_10_IP0 profile at Interoperability Point 0, level 4.0, Main tier, 10-bit content, non-monochrome, with 4:2:0 chroma subsampling co-located with (0, 0) luma sample, ITU-R BT.2100 color primaries, ITU-R BT.2100 PQ transfer characteristics, ITU-R BT.2100 YCbCr color matrix, and studio swing representation.

The string codecs="av02.0.01M.08" in this case would represent [AV2] Main_420_10_IP0 profile at Interoperability Point 0, level 2.1, Main tier, 8-bit content with 4:2:0 chroma subsampling, ITU-R BT.709 color primaries, transfer characteristics, matrix coefficients, and studio swing representation.

If any character that is not '.', digits, part of the AV2 4CC, or a tier value is encountered, the string SHALL be interpreted ignoring all the characters starting from that character.