Dolby Vision: Parse RPU in HEVC and AV1

[cjee21]

Parse RPU in HEVC which is needed for resolving #1482, #2118 and #2468

Also parse EL in BL+EL+RPU type HEVC streams.

Also parse RPU in AV1 metadata OBU.

[cjee21]

To resolve #2118, refer to 6.2 Annex II (page 23) of https://professionalsupport.dolby.com/s/article/What-is-Dolby-Vision-Profile?language=en_US

[cjee21]

Since it is complex to parse the RPU, I do not know if there are still bugs in the parsing.

I only parse the RPU and not fill or check integrity. I'll leave that to @JeromeMartinez since I do not know what info to extract and how to indicate enhancement layer type. The temporary section is to be removed when that information is filled and shown in output. I did not add any MEDIAINFO_TRACE guards either since that parsing is likely needed to resolve the two GitHub issues mentioned.

[cjee21]

Some interesting Dolby Vision IQ / Dolby Vision 2 stuff found in iPhone sample:

01189AC       ext_metadata_block (5 bytes)
01189AC        ext_block_length:                4 (0x4)
01189AC        ext_block_level:                 11 (0x0B) - (8 bits)
01189AD        ext_block_payload (4 bytes)
01189AD         content_type:                   4 (0x04) - (8 bits) - User Generated Content
01189AE         white_point:                    0 (0x0) - (4 bits) - D65
01189AF         reference_mode_flag:            No
01189AF         reserved:                       0 (0x0) - (3 bits)
01189AF         sharpness:                      0 (0x0) - (2 bits) - Default
01189AF         noise_reduction:                0 (0x0) - (2 bits) - Default
01189B0         mpeg_noise_reduction:           0 (0x0) - (2 bits) - Default
01189B0         frame_rate_conversion:          0 (0x0) - (2 bits) - Default
01189B0         brightness:                     0 (0x0) - (2 bits) - Default
01189B0         color:                          0 (0x0) - (2 bits) - Default
01189B1         reserved1:                      0 (0x0) - (2 bits)
01189B1         reserved2:                      0 (0x0) - (2 bits)

[JeromeMartinez]

Since it is complex to parse the RPU, I do not know if there are still bugs in the parsing.

Thank you, it helps a lot.

I'll leave that to @JeromeMartinez since I do not know what info to extract and how to indicate enhancement layer type.

I'll check (but only in October :( ), but do you have any hing about FEL vs MEL?

[cjee21]

do you have any hing about FEL vs MEL?

Hint? Find this part of the code added in this PR:

    // TEMPORARY ==========================================================================
    if (isMEL)
        Param2("EL Type", "Minimum Enhancement Layer (MEL)");
    else
        Param2("EL Type", "Full Enhancement Layer (FEL / non-MEL)");
    // end TEMPORARY ======================================================================

It is based on what's mentioned above in #2392 (comment).

[cjee21]

Hmm... Dolby sometimes calls RPU Reference Processing Unit and sometimes Reference Picture Unit...

Also...

Different Dolby Vision profiles use different mechanisms for carriage of the Dolby Vision reference processing unit. These include:
• Unspecified NAL units (NALu)
• T.35 versatile supplementary enhancement information (VSEI) using Dolby terminal provider code.
• T.35 Metadata open bitstream unit (OBU) using Dolby terminal provider code and wrapping the Dolby Vision reference processing unit in an EMDF

For now only parsed the Unspecified NALu type.

[JeromeMartinez]

T.35 versatile supplementary enhancement information (VSEI) using Dolby terminal provider code.

I never saw such content but if we find it, it would be easy to map the code to this one.

[cjee21]

T.35 Metadata open bitstream unit (OBU) using Dolby terminal provider code and wrapping the Dolby Vision reference processing unit in an EMDF

This one Dolby Vision profile 10 in AV1.

[JeromeMartinez]

This one Dolby Vision profile 10 in AV1.

I have missed something, link?

[cjee21]

This one Dolby Vision profile 10 in AV1.

I have missed something, link?

I have not looked into this but I saw it at the following places:
https://professionalsupport.dolby.com/s/article/What-is-Dolby-Vision-Profile?language=en_US
https://kodi.wiki/view/Samples
https://github.com/quietvoid/dovi_tool/blob/main/dolby_vision/src/av1/emdf.rs

[cjee21]

The L11 metadata part has mismatch between Dolby docs and Dolby patent so may have to tweak once there are more samples.

[JeromeMartinez]

The L11 metadata part has mismatch between Dolby docs and Dolby patent so may have to tweak once there are more samples.

dolby docs are usually at a later time and have fixes the patent doesn't have, so priority on Dolby docs.

[JeromeMartinez]

@cjee21 please split the PR, the 2 commits are 2 different things and I may merge the small one quickly.

[cjee21]

dolby docs are usually at a later time and have fixes the patent doesn't have, so priority on Dolby docs.

For the part that are in docs and sample files I follow that (docs did not say how many bits but it is parsed in a way that correct data is obtained from actual files) but some parts that I guess are used by Dolby Vision 2 are only seen in patent for now.

[cjee21]

Not sure if it was added but another useful thing for profile 7 would be MEL vs. FEL.

There is

Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0001 cd/m2, max: 1000 cd/m2
Dolby_Vision                             : RPU Present
 RPU_Profile                             : 1
 Base_Layer                              : 10 bits
 Enhancement_Layer                       : 10 bits
 VDR                                     : 12 bits
 Base_Layer_Video_Range                  : Limited
 Enhancement_Layer_Type                  : Full (FEL)
 Content_Mapping_version                 : 4.0
 Deduced_Profile                         : 7
Codec configuration box                  : hvcC+dvcC

[cjee21]

Dolby Vision also contains aspect ratio metadata that may be used to indicate cropping or active display area. But it is dynamic and may vary throughout the video so I am not sure if it should be shown in MediaInfo.

https://github.com/xbmc/xbmc/issues/27033#issuecomment-3497128236

[JeromeMartinez]

Dolby Vision also contains aspect ratio metadata that may be used to indicate cropping or active display area. But it is dynamic and may vary throughout the video so I am not sure if it should be shown in MediaInfo.

Please show this information, it is classic to have dynamic metadata (also for e.g. AVC aspect ratio), and we have a long term plan for better showing that when it changes, better to have the information now so it will be ready for metadata change display when we implement it.

[cjee21]

Please show this information

Rebased and done.

Using the faulty (double-cropped) sample from the xbmc/Kodi issue link above:

Width                                    : 3 840 pixels
Height                                   : 1 612 pixels
Display aspect ratio                     : 2.39:1
Active width                             : 3 840 pixels / 3 840 pixels / 3 840 pixels / 3 840 pixels / 3 840 pixels
Active height                            : 1 059 pixels / 1 059 pixels / 1 059 pixels / 1 059 pixels / 1 059 pixels
Active display aspect ratio              : 3.630

[JeromeMartinez]

@cjee21 It was reported to me that the code in this PR (or maybe a former version of this PR) is not exact and that the exact parsing is in FFmpeg.
https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/dovi_meta.h
https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/dovi_meta.c
https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/dovi_rpuenc.c
https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/dovi_rpudec.c
https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/dovi_rpu.h
https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/dovi_rpu.c

Could you crosscheck?
Maybe we could split in 2 steps:

• 1 PR (this one) with only the bitstream parsing hints you get from FFmpeg code.
• 1 (future) PR which adds stuff you found from other sources.

[cjee21]

@JeromeMartinez FFmpeg code changed recently: FFmpeg/FFmpeg@3474ec0

Is that the issue?

Looks like both this PR and FFmpeg is not accurate for that part now. Dolby seems to have repurposed those bits for Dolby Vision 2 motion control.

[cjee21]

Removed the uncertain parts. No data is better than potentially inaccurate data.

[quietvoid]

FWIW FFmpeg's L11 byte1 isn't right. It's supposed to be interpreted in LE, see quietvoid/dovi_tool@77b5402.

[cjee21]

So the bitstream in MSB first looks like:

content_type          u(8)
reserved              u(3)
reference_mode_flag   u(1)
whitepoint            u(4)

Correct? Is there a list of known white point values?

[quietvoid]

content_type u(8)
reserved u(3)
reference_mode_flag u(1)
whitepoint u(4)

Yes this works. I just prefer not doing it in reverse.

Is there a list of known white point values?

It seems to map like this according to the strings from Dolby's verifier.
Matches the info from https://professionalsupport.dolby.com/s/article/Dolby-Vision-IQ-Content-Type-Metadata-L11?language=en_US#white_point

0 = 6504K, D65
6879K
7254K
7629K
8004K
8379K
8754K
9129K
8 = 9504K, D93
9879K
10254K
10629K
11004K
11379K
11754K
12129K

[JeromeMartinez]

Is that the issue?

There was a delay between when a report was made and when I received it,, so it is a bit outdated because you changed it several times in the meantime.
In summary, with this PR FFmpeg shall be the reference, and any additions based on other sources shall be in a separate PR.
If you can separate the code between the different sources of information, that would make it easier for me to merge this PR.

[cjee21]

This PR was already based alot on FFmpeg from the beginning.

It was a mistake to follow FFmpeg L11 parsing which was wrong and is still wrong. This is corrected.

There is some strangeness in FFmpeg code so I do not trust it 100%.
Why el_bit_depth_minus8 is suddenly different from the other bit depths?

int bl_bit_depth_minus8 = get_ue_golomb_31(gb);
int el_bit_depth_minus8 = get_ue_golomb_long(gb);
int vdr_bit_depth_minus8 = get_ue_golomb_31(gb);

This part is inaccurate.

dm->l11.content_type = get_bits(gb, 8);
dm->l11.whitepoint = get_bits(gb, 4);
dm->l11.reference_mode_flag = get_bits1(gb);
skip_bits(gb, 3); /* reserved */
skip_bits(gb, 8); /* reserved */
skip_bits(gb, 8); /* reserved */

Every RPU in all DV samples I have are parsed properly. A randomly picked HEVC stream also produces a trace similar to output of Dovi tool. So I will not touch it anymore unless someone has a file that demonstrates a parsing issue.

I don't mind if this PR is reviewed last or if it is modified or not merged. There isn't much use for the typical consumer where DV files come mainly from iPhones and some rare Android phones. In these cases, the characteristics of the streams are already known except the newer L11 part that even FFmpeg or Dovi tool does not know.

[cjee21]

and for AV1 it is inside EMDF which is inside ITU T35 so it can wait after ITU T35 parser also.

[cjee21]

I just noticed sei_message_user_data_registered_itu_t_t35_B5_0031_GA94_09 looks similar to the metadata at the end of DV RPU and also contains active area offsets that are just skipped through at the moment.

[JeromeMartinez]

I just noticed sei_message_user_data_registered_itu_t_t35_B5_0031_GA94_09 looks similar to the metadata at the end of DV RPU and also contains active area offsets that are just skipped through at the moment.

I am not surprised :).
I am wondering what would be relevant to show (both SMPTE ST 2094-10 and DV).