New video manifest request

[CastagnaIT]

Check if this PR fulfills these requirements:

• My changes respect the Kodi add-on development rules
• I have read the CONTRIBUTING document.
• I made sure there wasn't another Pull Request opened for the same update/change
• I have successfully tested my changes locally

Types of changes

• New feature (non-breaking change which adds functionality)
• Feature change (non-breaking change which change behaviour of an existing functionality)
• Improvement (non-breaking change which improve functionality)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

There are new changes to the video manifest request, that require DRM info of the current session, the problem is always the same we do not know if the DRM info will be mandatory to add, this could be a big problem for the future of this addon...

---

Required info to add to netflix manifeste request:

• DRM session ID
• DRM key request (challenge)

Android case

These changes can be done in "easy way" for Android system case thanks to Kodi DRM interface
but currently is lack of a method to get the session ID.
[UPDATE] i have tried to add the support to get the session ID to Kodi interface (commit: CastagnaIT/xbmc@2ac2de1) but currently i have not tested need to build kodi android apk.

The only question is: the Kodi DRM interface provide a DRM session ID for crypting data, i suspect that it is different from the DRM session opened by ISA for video playback, this could be a problem? need testing (for HD content)

All other systems (Windows/Linux/Mac)

How to get DRM session info on these platforms?
1) Kodi provide a DRM interface but only for Android
2) InputStreamAdaptive do not allow to provide any DRM info and seem that the DRM session is constructed only after the manifest callback (if i understand is inside Session::InitializeDRM() method https://github.com/xbmc/inputstream.adaptive/blob/Matrix/src/main.cpp#L2258)

I have no idea on how to proceed, it is not easy situation (at least to me...i am not a widevine guru)
we can hope that netflix not cut out the old manifest request way (that we use) in short

Currently netflix also accepts to not specify the session ID, but we do not know if this option will be removed in the future...

I don't think i will be able to solve this situation on my own require changes to Kodi core and/or InputStreamAdaptive (opened Issue xbmc/inputstream.adaptive#666)
I will try to do some experiments i can't say any more at the moment

UPDATE 28/06/2021:

based on the suggestions i was able to make the necessary changes to ISA follow PR: xbmc/inputstream.adaptive#725
Tests on my ARM device are running at 1080P with success
but is needed tests also on android devices with L3 and L1
i can provide ISA binary to make tests on various platforms next days
all tests are ok!

In case of Feature change / Breaking change:

Describe the current behavior

Make the video manifest request as done until now could be obsolete in near future

Describe the new behavior

Make the video manifest request with DRM session info

Screenshots (if appropriate):

[jakermx]

Hi @CastagnaIT ,

What session info do you need to make your addon complaint for the new manifest?

I am not expert on DRM, but network and IT related, so I could get a solution.

;)

Cheers

[CastagnaIT]

is visible in the code changes:
the DRM key request (challenge) and the DRM session id

from what i understand to get these data, there should be an opportunity to initialise the DRM session before requesting the manifest, but currently ISA works differently and not allow it,
or else use the Kodi DRM interface but in this case is implemented for android only and we can get only the key request

[wagnerch]

@CastagnaIT I think what we are looking for is the following items to be exposed from the CDM Host interface:
SendPlatformChallenge
OnPlatformChallengeResponse

Somehow, back to a python binding. From what I could follow you initiate a platform challenge, which sends some data to the licensing server to be signed, and the CDM Host Interface executes the callback OnPlatformChallengeResponse with the signed data. I am not quite sure how this is invoked, was looking through Chromium source code to see, but it's a bit of a rabbit hole. Looking at ISA, it appears SendPlatformChallenge is a no-op function in the CdmAdapter.

What do you think? This stuff is a bit layered, so it's not easy to understand I guess without working with it. I don't know if logging Encrypted Media Extension calls in Chrome would help, but there is a logger extension. I also forget the scenario where this came up, wasn't it that we need to sign some bits of data before the manifest?

  // Structure provided to ContentDecryptionModule::OnPlatformChallengeResponse()
  // after a platform challenge was initiated via Host::SendPlatformChallenge().
  // All values will be NULL / zero in the event of a challenge failure.
  struct PlatformChallengeResponse {
    // |challenge| provided during Host::SendPlatformChallenge() combined with
    // nonce data and signed with the platform's private key.
    const uint8_t* signed_data;
    uint32_t signed_data_length;

    // RSASSA-PKCS1-v1_5-SHA256 signature of the |signed_data| block.
    const uint8_t* signed_data_signature;
    uint32_t signed_data_signature_length;

    // X.509 device specific certificate for the |service_id| requested.
    const uint8_t* platform_key_certificate;
    uint32_t platform_key_certificate_length;
  };

...

    // The following are optional methods that may not be implemented on all
    // platforms.

    // Sends a platform challenge for the given |service_id|. |challenge| is at
    // most 256 bits of data to be signed. Once the challenge has been completed,
    // the host will call ContentDecryptionModule::OnPlatformChallengeResponse()
    // with the signed challenge response and platform certificate. Size
    // parameters should not include null termination.
    virtual void SendPlatformChallenge(const char* service_id,
      uint32_t service_id_size,
      const char* challenge,
      uint32_t challenge_size) = 0;

...

    // Called by the host after a platform challenge was initiated via
    // Host::SendPlatformChallenge().
    virtual void OnPlatformChallengeResponse(
      const PlatformChallengeResponse& response) = 0;

[CastagnaIT]

i have updated the description on top with my updates,

but SendPlatformChallenge seem used to send, not to get the challenge,
IMO i think we need to initialize the DRM (to get the drm info) before that we make the manifest request in the addon (that will be forwarded to ISA)
I was thinking something like this:

->User run video play
->Kodi talk to ISA to run the video
->ISA initialize the DRM before send the Manifest callback to the addon
->ISA perform the Manifest HTTP request callback to the addon by sending also the challenge and the session ID (in the request)
->We have all the DRM data to perform the our manifest request to netflix server

but i do not know how to make this (if possible)

[wagnerch]

but SendPlatformChallenge seem used to send, not to get the challenge,

OnPlatformChallengeResponse would return a signature of the platform challenge. I think this is the issue we had with the 540p on ARM, and when I ran mitm-proxy on RPi we found it sent back a payload with CDM platform data that Netflix required to playback 1080p.

Do we have any reference what this data looks like? Because in the case of issue #655 I am pretty sure that payload was the platform challenge. And I think SendPlatformChallenge generates part of the payload, but the other part of input is a service_id and challenge which probably come from Netflix, and that message is sent to the Widevine CDM server for signing and returned via OnPlatformChallengeResponse the signed data, signature, and certificate, and sent back to Netflix which at that point they could verify the signature and the service_id/challenge data matches what they sent.

I tried tracing through the DRM Interface, it's a rabbit hole that calls JNI-based code in Android called MediaDrm. You can read more about that at:
https://developer.android.com/reference/android/media/MediaDrm

[wagnerch]

Hmm, so I think I see what your saying. I was looking back at the payload captured with mitm, and it is the same payload/size of the ".challenge" file that ISA writes out to the profile folder. It's curious, how Android does this when the CDM interface doesn't seem to support it.

The session in the CDM layer is initiated by CreateSessionAndRequest, and it bubbles back to OnSessionMessage to set the session id & challenge. In order for that function to fire (which is called from WV_CencSingleSampleDecrypter constructor) it requires pssh and key id, as mentioned in the ISA ticket.

So I think it comes down to figuring out how Android does this, and how RPi4 / ChromeOS does this. I think we should see something in the EME Logger (https://developers.google.com/web/updates/2015/09/eme-logger), since as I understand it is Netflix player has to work through EME (Encrypted Media Extensions) to access the CDM layer. When I play a video, it seems like it does initiate multiple sessions through EME, so maybe if we understand what's going on there it might give a clue on how the CDM layer needs to be called in ISA.

[glennguy]

The challenge is generated by widevinecdm at some point after initialization and is captured in this callback here:
https://github.com/xbmc/inputstream.adaptive/blob/c96342507c409319666a0455ec069ba1a80d2c0d/wvdecrypter/wvdecrypter.cpp#L391, and you're correct @wagnerch it isn't part of the Android implementation in ISA.

[wagnerch]

@glennguy I noticed with ISA you offer an initial manifest fetch which afterwards CreateSessionAndGenerateRequest is called and at that point the session ID & challenge is available, so I am wondering if this should align to prefetch/manifest in Netflix. And then ISA offers inputstream.adaptive.manifest_update_parameter which could be a URL and I wonder if at this point you would call licensedManifest in Netlifx passing the established session id & challenge.

What I don't understand is if the first chunk (video segment) is already started at this point. In other words, perhaps the first chunk is low res, but subsequent chunks would be 1080p with the updated manifest. @CastangaIT not sure if this would solve the problem, but it would seem to me for Android you can probably go directly to licensedManifest but on other platforms maybe it's 2-step process. I know on ARM devices the problem is if we don't have the challenge then the manifest will not report 1080p chunks.

The basic idea is whether we can have ISA prefetch the manfiest to establish the session & challenge, and then immediately after request a manifest update passing the session & challenge and then parsing the manifest to start downloading & playing segments.

[CastagnaIT]

@wagnerchthe do not take in account of the current "prefetch/manifest" we will use only "licensedManifest" type (the website not use anymore the old version except for some non-drm videos),
IMO a double manifest request not sound good to me in my experience with MSL exceptions suspect this could create other future problems, perhaps the glennguy way of "bare-bones init version" is perhaps bit better

[wagnerch]

IMO a double manifest request not sound good to me in my experience with MSL exceptions suspect this could create other future problems, perhaps the glennguy way of "bare-bones init version" is perhaps bit better

@CastagnaIT agreed. So have you figured out on which platforms/videos require this new manifest request? I guess what I can do is, if it still works on the RPI4, is run the proxy (netflix-mitm-proxy) plus EME Logger and see what happens there, if that will be helpful. The reason I bring up this idea, even on RPi4 when I bring up a direct /watch/ url in the browser I can see 3 separate calls to GenerateSessionCall, with the same pssh and 3 different session id's as output. So I don't know if that is to deal with this problem, or it's prefetching previews, or what is happening there. There is also 3 MessageEvent, with 3 separate license-request message type (which is what we are calling "challenge" in the manifest request).

[wagnerch]

Looks like netflix-mitm-proxy isn't working on Windows w/ Firefox (get a error when trying to play a video). But I do see it's using licensedManifest.

[wagnerch]

Think I might have something. So Netflix actually pre-generates the pssh initialization data and embeds it in cadmium-player.js, so what they are doing is an EME generateRequest, pass the license data into the licensedManifest MSL call, and then generating the MPD and playing the video.

So we would need something from ISA to pre-call CreateSessionAndGenerateRequest with our pre-generated pssh initialization data, and then generate the manifest and resume as normal.

This is what it looks like in cadmium player, the c.prototype.tyb is deciding based on (r) passed into the promise resolve (which is an integer 0, 1, 2 that indicates the DRM type playready, widevine, etc), and returns the pre-generated initialization data used by generateRequest.

The real bear is how the heck do you parse this out, is it static for everyone, or is this JS file dynamically generated for everyone?

EDIT: Ycb is the one for widevine. Maybe searching for "AAAANHBzc2g", I think that portion is a fixed header.

00000000 00 00 00 34 70 73 73 68 00 00 00 00 ed ef 8b a9 |...4pssh........|
00000010 79 d6 4a ce a3 c8 27 dc d5 1d 21 ed 00 00 00 14 |y.J...'...!.....|
00000020 08 01 12 10 00 00 00 00 03 d2 67 49 00 00 00 00 |..........gI....|
00000030 00 00 00 00 |....|

        c.prototype.tyb = function () {
          var p;
          p = this;
          return this.il.Lr().then(function (r) {
            switch (r) {
              case m.Em.PD:
                return n.U8a;
              case m.Em.XA:
                return n.Abb;
              default:
                return n.Ycb;
            }
          }).then(function (r) {
            return r.map(p.pc.decode);
          });
        };
      function (q, b) {
        Object.defineProperty(b, '__esModule', {
          value: !0
        });
        b.Ycb = [
          'AAAANHBzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAABQIARIQAAAAAAPSZ0kAAAAAAAAAAA=='
        ];
        b.U8a = [
          'c2tkOi8vbmV0ZmxpeC9BQUFBQkFBQUFBQUV3MzFBcDVKNTB0Qml4WTVMaTNYSGJFc3dKdDdybTdhdEExVWpPMkw3Q1ArckJYTT0='
        ];
        b.Abb = [
          'AAADPHBzc2gAAAAAmgTweZhAQoarkuZb4IhflQAAAxwcAwAAAQABABIDPABXAFIATQBIAEUAQQBEAEUAUgAgAHgAbQBsAG4AcwA9ACIAaAB0AHQAcAA6AC8ALwBzAGMAaABlAG0AYQBzAC4AbQBpAGMAcgBvAHMAbwBmAHQALgBjAG8AbQAvAEQAUgBNAC8AMgAwADAANwAvADAAMwAvAFAAbABhAHkAUgBlAGEAZAB5AEgAZQBhAGQAZQByACIAIAB2AGUAcgBzAGkAbwBuAD0AIgA0AC4AMgAuADAALgAwACIAPgA8AEQAQQBUAEEAPgA8AFAAUgBPAFQARQBDAFQASQBOAEYATwA+ADwASwBJAEQAUwA+ADwASwBJAEQAIABBAEwARwBJAEQAPQAiAEEARQBTAEMAVABSACIAIABWAEEATABVAEUAPQAiAEEAQQBBAEEAQQBNAFkARQB4AEkARQBBAEEAQQBBAEEAQQBBAEEAQQBBAEEAPQA9ACIAPgA8AC8ASwBJAEQAPgA8AC8ASwBJAEQAUwA+ADwALwBQAFIATwBUAEUAQwBUAEkATgBGAE8APgA8AEwAQQBfAFUAUgBMAD4AaAB0AHQAcAA6AC8ALwBjAGEAcABwAHIAcwB2AHIAMAA2AC8AcwBpAGwAdgBlAHIAbABpAGcAaAB0ADUALwByAGkAZwBoAHQAcwBtAGEAbgBhAGcAZQByAC4AYQBzAG0AeAA8AC8ATABBAF8AVQBSAEwAPgA8AEwAVQBJAF8AVQBSAEwAPgBoAHQAdABwADoALwAvAGMAYQBwAHAAcgBzAHYAcgAwADYALwBzAGkAbAB2AGUAcgBsAGkAZwBoAHQANQAvAHIAaQBnAGgAdABzAG0AYQBuAGEAZwBlAHIALgBhAHMAbQB4ADwALwBMAFUASQBfAFUAUgBMAD4APABEAEUAQwBSAFkAUABUAE8AUgBTAEUAVABVAFAAPgBPAE4ARABFAE0AQQBOAEQAPAAvAEQARQBDAFIAWQBQAFQATwBSAFMARQBUAFUAUAA+ADwALwBEAEEAVABBAD4APAAvAFcAUgBNAEgARQBBAEQARQBSAD4A'
        ];
      },

[wagnerch]

https://www.w3.org/TR/eme-initdata-cenc/#common-system

I think if I understand this we have to parse it from JS. Because there is at least one KID embedded in there. Perhaps we have to look for an RE "['[^']+']" and parse each B64 payload looking for pssh w/ SystemID for Widevine?

var pssh = [
    0x00, 0x00, 0x00, 0x4c, 0x70, 0x73, 0x73, 0x68, // BMFF box header (76 bytes, 'pssh')
    0x01, 0x00, 0x00, 0x00,                         // Full box header (version = 1, flags = 0)
    0x10, 0x77, 0xef, 0xec, 0xc0, 0xb2, 0x4d, 0x02, // SystemID
    0xac, 0xe3, 0x3c, 0x1e, 0x52, 0xe2, 0xfb, 0x4b,
    0x00, 0x00, 0x00, 0x02,                         // KID_count (2)
    0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, // First KID ("0123456789012345")
    0x38, 0x39, 0x30, 0x31, 0x32, 0x33, 0x34, 0x35,
    0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, // Second KID ("ABCDEFGHIJKLMNOP")
    0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50,
    0x00, 0x00, 0x00, 0x00,                         // Size of Data (0)
];

[wagnerch]

Here is a kludgy routine to scavenge the pssh from cadmium-player:

#!/usr/bin/env python3
import sys
import re
import base64

f = open('cadmium-playercore-6.0030.049.911.js', 'r')
content = f.read()
f.close()
found = re.compile('\["([^"]+)"\];', re.DOTALL).findall(content)
for f in found:
    try:
        d = base64.b64decode(f)
        if d[4:8] == b'pssh' and d[12:28] == bytes([0xed, 0xef, 0x8b, 0xa9, 0x79, 0xd6, 0x4a, 0xce, 0xa3, 0xc8, 0x27, 0xdc, 0xd5, 0x1d, 0x21, 0xed]):
            print("FOUND: " + f)
    except:
#        print(sys.exc_info())
        pass

also, here is a Pssh box implementation in Python that can parse this binary data.

https://github.com/google/shaka-packager/blob/master/packager/tools/pssh/pssh-box.py

[CastagnaIT]

@wagnerch wow you have found many things
the pssh is hardcoded in js so for now we can hardcode in the addon, we can add the parser from website in later time if will be needed

looking at the format specifications for pssh seem that on ISA we have already one parser that can be copied and reused, here:
https://github.com/xbmc/inputstream.adaptive/blob/Matrix/src/parser/HLSTree.cpp#L136-L144
seem to look the same thing
in this way we have only to send the pssh to ISA, ISA extract the KID to initialize the drm and we can go

[wagnerch]

in this way we have only to send the pssh to ISA, ISA extract the KID to initialize the drm and we can go

Yeah, I was looking at this yesterday, so the basic call flow through ISA is:

-> CInputStreamAdaptive::Open
-> Session::Initialize
-> Session::GetSupportedDecrypterURN (this is where the SSD is loaded, which is the wvdecrypter.cpp)
-> adaptive::DASHTree::open
-> Session::InitializePeriod
-> Session::InitializeDRM
-> WVDecrypter::OpenDRMSystem
-> WVDecrypter::CreateSingleSampleDecrypter

What needs to happen is WVDecrypter::OpenDRMSystem has to be moved earlier, to just after Session::GetSupportedDecrypterURN, I didn't fully flush out dependencies but I think that can be cleanly moved. But WVDecyrpter::OpenDRMSystem is what instantiates the CDM adapter for Widevine.

The next problem is splitting apart or copying bits of WVDecrypter::CreateSingleSampleDecrypter && WV_CencSingleSampleDecrypter::WV_CencSingleSampleDecrypter. The constructor of WV_CencSingleSampleDecrypter would need to be changed and moved before adaptive::DASHTree::open, because adaptive::DASHTree::open is where the manifest callback URL is opened and parsed. But this part of the change would only occur if we pass in that initialization data.

There is a lot of interconnect bits to look after, so I don't know how easy it is to change. But getting the additional list item property in is easy, it's the moving up of creating the session & request via CDM that is tricky. Based on the code it also does re-keying of the pssh. So I am guessing what should happen is we pass in the pssh, it is used for the initial CreateSessionAndGenerateRequest, and then it will call the callback manifest url passing the license data & session id as arguments to the GET, which we would embed in the licensedManifest convert to DASH MPD form which includes the pssh again (maybe different?), and then ISA would treat that as a re-key if the session is already created -- if it isn't already created then it would setup the session, and life goes on as normal.

There are 4 or so major components in ISA:

1. Addon interface
2. SSD interface (dynamic lib, this is where the WV decrypter interface lives)
3. CDM adapter/interface
4. Parser interface (HLS/DASH/etc))

[CastagnaIT]

i agree, i also suspect that it may be necessary to update with new pssh (of the converted manifest)

[CastagnaIT]

i have tried to do some experiment with ISA,
and i don't even know if I'm doing the right things
in any case i am stuck on the missing KID

i have tried to use pssh-box.py but raise an error and not parse the PSSH

then i have converted the PSSH base 64
AAAANHBzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAABQIARIQAAAAAAPSZ0kAAAAAAAAAAA==
to HEX and i have tried to compare with the W3 documentation
https://www.w3.org/TR/eme-initdata-cenc/#example
to see what happen and i see some incoherence? i think this is the problem that cause the pssh-box.py to fails

0x00 0x00 0x00 0x34 0x70 0x73 0x73 0x68 // BMFF box header (76 bytes, 'pssh')
0x00 0x00 0x00 0x00                     // Full box header (version = 1, flags = 0)
0xED 0xEF 0x8B 0xA9 0x79 0xD6 0x4A 0xCE // SystemID
0xA3 0xC8 0x27 0xDC 0xD5 0x1D 0x21 0xED
0x00 0x00 0x00 0x14 0x08 0x01 0x12 0x10 0x00 0x00 0x00 0x00 0x03 0xD2 0x67 0x49 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00

Last line should cointains the sections: KID_count, KID data (one or more), Size of Data
but the KID_count section seem strange, i am not sure what to do to extract the KID

[wagnerch]

@CastagnaIT My impression of this spec is it's for a specific system id ("common" system id). I don't know the details, but Widevine doesn't follow this specific spec document you referenced. Widevine's PSSH Box version is 0, it doesn't have Key ID's in this format. Instead you have to parse the PSSH data payload, and run that through a protobuf parser to get the result.

1. You need to compile this file: https://github.com/google/shaka-packager/blob/master/packager/media/base/widevine_pssh_data.proto with protoc (protoc --python_out=. widevine_pssh_data.proto)

2. You need to parse the PSSH box data out of the base64 string (the parser for this bit is a little more elegant at https://github.com/google/shaka-packager/blob/56e227267c9091a0f65b4d92d9064dda4557f3a7/packager/tools/pssh/pssh-box.py#L237):
   uint32 (big endian) - pssh size
   uint8 (4 bytes) - "pssh" magic
   uint32 (big endian) - version & flags, 8 LSB is version, 24 MSB is flags (no idea what the flags are)
   uint8 (16 bytes) - System ID UUID
   uint32 (big endian) - PSSH data size
   uint8 ([PSSH data size]) - PSSH data

3. PSSH data is then passed into protobuf, wv = widevine_pssh_data_pb2.WidevinePsshData(), wv.ParseFromString()

4. if wv.key_id, then you have 1 or more key id's (array).

You can see that in action here:
https://github.com/google/shaka-packager/blob/56e227267c9091a0f65b4d92d9064dda4557f3a7/packager/tools/pssh/pssh-box.py#L174

This is what I get:

PSSH Box v0
  System ID: edef8ba9-79d6-4ace-a3c8-27dcd51d21ed
  PSSH data (size: 20): 080112100000000003d267490000000000000000
  Key IDs (1):
    00000000-03d2-6749-0000-000000000000
Widevine PSSH found -- AAAANHBzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAABQIARIQAAAAAAPSZ0kAAAAAAAAAAA==

Here is the hack script I am using:

#!/usr/bin/env python3
import re
import base64
import uuid
import glob
import os
import widevine_pssh_data_pb2

WIDEVINE_SYSTEM_ID = uuid.UUID('edef8ba9-79d6-4ace-a3c8-27dcd51d21ed')

for js in sorted(glob.glob(os.getcwd() + '/cadmium-playercore-*.js')):
    print(js + ":")
    with open(js, 'r') as f:
        cadmium_playercore_js = f.read()

    haystack = re.compile('=\s*\["([^"]+)"\];', re.DOTALL).findall(cadmium_playercore_js)
    for needle in haystack:
        try:
            ## parse pssh data
            init_data = base64.b64decode(needle)

            off = 0
            size = int.from_bytes(init_data[off:off + 4], "big")
            off += 4
            if size != len(init_data):
                raise Exception('Invalid number of bytes')

            box_type = init_data[off:off + 4]
            off += 4
            if box_type != b'pssh':
                raise Exception('Invalid box type')

            version_and_flags = int.from_bytes(init_data[off:off + 4], "big")
            off += 4
            version = version_and_flags >> 24
            if version > 1:
                raise Exception('Invalid PSSH version %d' % version)

            system_id = uuid.UUID(bytes=init_data[off:off + 16])
            off += 16

            key_ids = []
            if version > 1:
                count = int.from_bytes(init_data[off:off + 4], "big")
                off += 4
                while count > 0:
                    key_ids.append(uuid.UUID(bytes=init_data[off:off + 16]))
                    off += 16
                    count =- 1

            pssh_data_size = int.from_bytes(init_data[off:off + 4], "big")
            off += 4
            pssh_data = init_data[off:]
            if pssh_data_size != len(pssh_data):
                raise Exception('Invalid PSSH data size')

            print('PSSH Box v%d' % version)
            print('  System ID: %s' % system_id)
            if version == 1:
                print('  Key IDs (%d):' % len(key_ids))
                for key in key_ids:
                    print('    %s' % key)
            print('  PSSH data (size: %d): %s' % (pssh_data_size, pssh_data.hex()))

            ## if we got here, then it's a valid pssh, check if it is
            ## widevine
            if WIDEVINE_SYSTEM_ID == system_id:
                wv = widevine_pssh_data_pb2.WidevinePsshData()
                wv.ParseFromString(pssh_data)
                if wv.key_id:
                    print('  Key IDs (%d):' % len(wv.key_id))
                    for k in wv.key_id:
                        print('    %s' % uuid.UUID(bytes=k))

                if wv.HasField('provider'):
                    print('  Provider: %s' % wv.provider)

                if wv.HasField('content_id'):
                    print('  Content ID: %s' % base64.b16encode(wv.content_id).decode())

                if wv.HasField('policy'):
                    print('  Policy: %s' % wv.policy)

                if wv.HasField('crypto_period_index'):
                    print('  Crypto Period Index: %d' % wv.crypto_period_index)

                if wv.HasField('protection_scheme'):
                    print('  Protection Scheme: %s' % struct.pack('>L', wv.protection_scheme))

                print("Widevine PSSH found -- %s" % needle)
                break
        except Exception as error:
#            print("-- " + str(error))
            pass

    print("")

And here is the compiled protobuf to Python,
widevine_pssh_data_pb2.py:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: widevine_pssh_data.proto

import sys
_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)

_sym_db = _symbol_database.Default()




DESCRIPTOR = _descriptor.FileDescriptor(
  name='widevine_pssh_data.proto',
  package='shaka.media',
  syntax='proto2',
  serialized_options=None,
  serialized_pb=_b('\n\x18widevine_pssh_data.proto\x12\x0bshaka.media\"\x8f\x02\n\x10WidevinePsshData\x12:\n\talgorithm\x18\x01 \x01(\x0e\x32\'.shaka.media.WidevinePsshData.Algorithm\x12\x0e\n\x06key_id\x18\x02 \x03(\x0c\x12\x10\n\x08provider\x18\x03 \x01(\t\x12\x12\n\ncontent_id\x18\x04 \x01(\x0c\x12\x0e\n\x06policy\x18\x06 \x01(\t\x12\x1b\n\x13\x63rypto_period_index\x18\x07 \x01(\r\x12\x17\n\x0fgrouped_license\x18\x08 \x01(\x0c\x12\x19\n\x11protection_scheme\x18\t \x01(\r\"(\n\tAlgorithm\x12\x0f\n\x0bUNENCRYPTED\x10\x00\x12\n\n\x06\x41\x45SCTR\x10\x01\"G\n\x0eWidevineHeader\x12\x0f\n\x07key_ids\x18\x02 \x03(\t\x12\x10\n\x08provider\x18\x03 \x01(\t\x12\x12\n\ncontent_id\x18\x04 \x01(\x0c')
)



_WIDEVINEPSSHDATA_ALGORITHM = _descriptor.EnumDescriptor(
  name='Algorithm',
  full_name='shaka.media.WidevinePsshData.Algorithm',
  filename=None,
  file=DESCRIPTOR,
  values=[
    _descriptor.EnumValueDescriptor(
      name='UNENCRYPTED', index=0, number=0,
      serialized_options=None,
      type=None),
    _descriptor.EnumValueDescriptor(
      name='AESCTR', index=1, number=1,
      serialized_options=None,
      type=None),
  ],
  containing_type=None,
  serialized_options=None,
  serialized_start=273,
  serialized_end=313,
)
_sym_db.RegisterEnumDescriptor(_WIDEVINEPSSHDATA_ALGORITHM)


_WIDEVINEPSSHDATA = _descriptor.Descriptor(
  name='WidevinePsshData',
  full_name='shaka.media.WidevinePsshData',
  filename=None,
  file=DESCRIPTOR,
  containing_type=None,
  fields=[
    _descriptor.FieldDescriptor(
      name='algorithm', full_name='shaka.media.WidevinePsshData.algorithm', index=0,
      number=1, type=14, cpp_type=8, label=1,
      has_default_value=False, default_value=0,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='key_id', full_name='shaka.media.WidevinePsshData.key_id', index=1,
      number=2, type=12, cpp_type=9, label=3,
      has_default_value=False, default_value=[],
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='provider', full_name='shaka.media.WidevinePsshData.provider', index=2,
      number=3, type=9, cpp_type=9, label=1,
      has_default_value=False, default_value=_b("").decode('utf-8'),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='content_id', full_name='shaka.media.WidevinePsshData.content_id', index=3,
      number=4, type=12, cpp_type=9, label=1,
      has_default_value=False, default_value=_b(""),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='policy', full_name='shaka.media.WidevinePsshData.policy', index=4,
      number=6, type=9, cpp_type=9, label=1,
      has_default_value=False, default_value=_b("").decode('utf-8'),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='crypto_period_index', full_name='shaka.media.WidevinePsshData.crypto_period_index', index=5,
      number=7, type=13, cpp_type=3, label=1,
      has_default_value=False, default_value=0,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='grouped_license', full_name='shaka.media.WidevinePsshData.grouped_license', index=6,
      number=8, type=12, cpp_type=9, label=1,
      has_default_value=False, default_value=_b(""),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='protection_scheme', full_name='shaka.media.WidevinePsshData.protection_scheme', index=7,
      number=9, type=13, cpp_type=3, label=1,
      has_default_value=False, default_value=0,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
  ],
  extensions=[
  ],
  nested_types=[],
  enum_types=[
    _WIDEVINEPSSHDATA_ALGORITHM,
  ],
  serialized_options=None,
  is_extendable=False,
  syntax='proto2',
  extension_ranges=[],
  oneofs=[
  ],
  serialized_start=42,
  serialized_end=313,
)


_WIDEVINEHEADER = _descriptor.Descriptor(
  name='WidevineHeader',
  full_name='shaka.media.WidevineHeader',
  filename=None,
  file=DESCRIPTOR,
  containing_type=None,
  fields=[
    _descriptor.FieldDescriptor(
      name='key_ids', full_name='shaka.media.WidevineHeader.key_ids', index=0,
      number=2, type=9, cpp_type=9, label=3,
      has_default_value=False, default_value=[],
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='provider', full_name='shaka.media.WidevineHeader.provider', index=1,
      number=3, type=9, cpp_type=9, label=1,
      has_default_value=False, default_value=_b("").decode('utf-8'),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
    _descriptor.FieldDescriptor(
      name='content_id', full_name='shaka.media.WidevineHeader.content_id', index=2,
      number=4, type=12, cpp_type=9, label=1,
      has_default_value=False, default_value=_b(""),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      serialized_options=None, file=DESCRIPTOR),
  ],
  extensions=[
  ],
  nested_types=[],
  enum_types=[
  ],
  serialized_options=None,
  is_extendable=False,
  syntax='proto2',
  extension_ranges=[],
  oneofs=[
  ],
  serialized_start=315,
  serialized_end=386,
)

_WIDEVINEPSSHDATA.fields_by_name['algorithm'].enum_type = _WIDEVINEPSSHDATA_ALGORITHM
_WIDEVINEPSSHDATA_ALGORITHM.containing_type = _WIDEVINEPSSHDATA
DESCRIPTOR.message_types_by_name['WidevinePsshData'] = _WIDEVINEPSSHDATA
DESCRIPTOR.message_types_by_name['WidevineHeader'] = _WIDEVINEHEADER
_sym_db.RegisterFileDescriptor(DESCRIPTOR)

WidevinePsshData = _reflection.GeneratedProtocolMessageType('WidevinePsshData', (_message.Message,), dict(
  DESCRIPTOR = _WIDEVINEPSSHDATA,
  __module__ = 'widevine_pssh_data_pb2'
  # @@protoc_insertion_point(class_scope:shaka.media.WidevinePsshData)
  ))
_sym_db.RegisterMessage(WidevinePsshData)

WidevineHeader = _reflection.GeneratedProtocolMessageType('WidevineHeader', (_message.Message,), dict(
  DESCRIPTOR = _WIDEVINEHEADER,
  __module__ = 'widevine_pssh_data_pb2'
  # @@protoc_insertion_point(class_scope:shaka.media.WidevineHeader)
  ))
_sym_db.RegisterMessage(WidevineHeader)


# @@protoc_insertion_point(module_scope)

[CastagnaIT]

yes i had compiled the whole package shaka
very thank you for the reworked extractor!! it works

I was able to continue making changes to ISA and at first glance I managed to initialize the DRM session and get the session id and the challenge value to pass in the manifest proxy callback, seem without ripercussions to the playback

but i will need to test on the ARM device to see if this changes really works to get the 1080P resolutions
crossfingers

[CastagnaIT]

@wagnerch i am trying to add a new property to parse here:
https://github.com/xbmc/inputstream.adaptive/blob/2.6.17-Matrix/src/main.cpp#L3336-L3405
of course i have added the same new property in the python play callback side
that i have called as inputstream.adaptive.preinit_drm

but the new property is not listed from GetProperties()
do you know if you need to modify Kodi as well to pass new ListItem properties to ISA?

[CastagnaIT]

solved found in the addon.xml on key listitemprops

[CastagnaIT]

after many hours of f##king building problems with ISA binary
i finally managed to test it on an ARM device with CoreELEC and drumroll
...
it's working!!🎉
i can get and watch the videos at 1080P !

I will need to improve the changes on ISA before making a draft PR,
and in any case i will need to ask how to better handle some points

I will also have to do tests on all other operative systems

[vascobraga41]

Is there any way to test this changes? My arm devices all dropped to playing 540p resolution limitation when before last week they all played 1080p without any problem.

[vascobraga41]

I'll try later, just a question, is it normal to ESN be just blank? I messed with it because it was, at first.

[vascobraga41]

my fault, if you delete "cdm/widevine" folder wv stop working until a device reboot,
now on my device works as before...

please try to do this steps:

1. try reinstall widevine, reboot device, try play
2. if step 1 not works, update ISA: isa-libreelec-armv8-matrix-removeddispose.zip, reboot device, try play

do not touch the ESN the problem is with widevine/ISA not netflix add-on

Did some quick tests on my VIM3 and reinstalling widevine didn't work. Tried your new ISA and on the first reboot it disappeared. Reinstalled and rebooted and the first video plays always. The second throws the same error. I'll produce logs later tonight.

[betatester3016]

@CastagnaIT I had debug enabled in NF addon but not in Kodi.

Here is updated log: http://ix.io/3rGD ISA & NF from #1129 < ERROR

The 4 different scenarios were provided as a frame of reference.

[vascobraga41]

Reinstallwidevine.log
NewISA.log
Same problem both times, it plays the first video and not the second. Just a note, it installed old widevine so I'll update to latest nightly and repeat with new widevine

Now with latest widevine. It plays one time (not always) and then it stops playing.
ReinstallWidevineNew.log
NewISANewWidevine.log

[CastagnaIT]

thanks for the report,
today i also my device finally has raised same error
i am not sure but could be due to multithread concurrency with the variable used try this:
REMOVED
EDIT: is happened again... the value is still lost

[vascobraga41]

Can you share some wip ISA that works partially @CastagnaIT ?

[betatester3016]

@CastagnaIT for what it is worth: I can run tests and benchmarks on several ARM devices from SoC S905X to S905X3.

[CastagnaIT]

ATM other tests are not needed
is needed to find what is wrong in ISA PR code
i am not able to understand the cause that blank data is sent by curl while the data exists

[CastagnaIT]

Finally thanks to glennguy seem i have fixed the bug
i have successfully played 6 videos

fixed files:
PATCHED ADDON: plugin.video.netflix-matrix-t5.zip

ISA CoreELEC ARM: isa-linux-armv8-matrix-fixed.zip

if yours feedback are good i will recompile fixed ISA for the others operative systems
PS: after updated of course remember to reboot

[vascobraga41]

Just did a quick test with 4 videos and all played without problems at 1080p. Seems fine to me but will test more during the day.

[CastagnaIT]

i have fixed another problem, these are all the definitive builds:

inputstream.adaptive-coreelec-armv8-matrix.zip
inputstream.adaptive-windows-x86_64-matrix-ac2f53a.zip
inputstream.adaptive-linux-x86_64-matrix-ac2f53a.zip
inputstream.adaptive-android-aarch64-matrix-ac2f53a.zip
inputstream.adaptive-android-armv7-matrix-ac2f53a.zip

plugin.video.netflix_1.16.1+matrix.1_20210704.zip

PS: On linux / coreelec / android remember to reboot the device after updated the addons

mainly i need someone test of CoreELEC ARM device and Android (Widevine L1) with 4K support

[vpeter4]

I tried on CoreELEC nightly 10 times of play/stop and all good with 1080p.

[betatester3016]

@CastagnaIT

I tested thoroughly on SoC S905X & S905X3 (lowest and highest SoC supported by CE Matrix) running CoreELEC Matrix 19.2 RC2

The combination of NF and IS.A for CoreELEC Matrix:

inputstream.adaptive-coreelec-armv8-matrix.zip
plugin.video.netflix_1.16.1+matrix.1_20210704.zip

works on both test devices.

Here are screenshot and log from S905X3 device for your reference.

[vascobraga41]

I tried in the Firestick but the ISA version isn't compatible.

[CastagnaIT]

@vascobraga41 i have add in the list android-armv7 zip version i hope this works
@vpeter4 @betatester3016 thanks we can say that for ARM devices is full working

[vpeter4]

@CastagnaIT: You don't have Kodi freezes anymore?

[CastagnaIT]

@vpeter4 I didn't have time to do any more tests

[vascobraga41]

@CastagnaIT i can confirm that in firestick i get the videos in Dolby Vision, 4k, etc and also compared with the tv app and the resolutions match. So I think android is also solved.

[CastagnaIT]

thank you perfect

[notoco]

I can confirm. It works. Coreelec last nighthly. Well done.