[node] flatpak-node-generator.py failed to deal with deps with direct git reference

[proletarius101]

What is a dependency with direct git reference?

https://github.com/standardnotes/desktop/blob/3fbb58adf16a564d6773aae6cce273325f9eeba8/package.json#L12

  "dependencies": {
    ...
    "decrypt": "github:standardnotes/decrypt#master",
    ...
  },

What did I do?

python flatpak-node-generator.py npm ../standardnotes-desktop/package-lock.json --xdg-layout  -r

The results

npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/@standardnotes%2fsncrypto-web failed: cache mode is 'only-if-cached' but no cached response available.

A complete log could be found at https://flathub.org/builds/#/builders/35/builds/4478

What I can suggest

crypto-web is a dependency of github:standardnotes/decrypt#master. I think it's failed to be included in the cache.

Related PR

flathub/flathub#2086

[proletarius101]

Some interesting logs:

21 silly removeObsoleteDep removing decrypt@github:standardnotes/decrypt#5d487038d3568d180e7cf561d774e1b685985d62 from the tree as its been replaced by a newer version or is no longer required
22 silly removeObsoleteDep removing @standardnotes/sncrypto-web@1.2.9 from the tree as its been replaced by a newer version or is no longer required
23 silly removeObsoleteDep removing @standardnotes/sncrypto-common@1.2.9 from the tree as its been replaced by a newer version or is no longer required
24 silly removeObsoleteDep removing libsodium-wrappers@0.7.8 from the tree as its been replaced by a newer version or is no longer required
25 silly removeObsoleteDep removing libsodium@0.7.8 from the tree as its been replaced by a newer version or is no longer required
26 silly removeObsoleteDep removing @standardnotes/snjs@2.0.22 from the tree as its been replaced by a newer version or is no longer required

[proletarius101]

So after further investigation, a more elaborated reason is, the git-hosted module has a version, which is obvious, however, it's not tagged. npm tries to replace it with @standardnotes/sncrypto-web@1.2.9. That's why it can't use the cache

[catsout]

First you should use git source instead of tar.gz, as it has submodules.
For npm git dependence try this, I try to fix it and not merged to here now.

You may need add env npm_config_offline: 'true' and npm_config_no_save: 'true', --no-save for check package{-lock}.json file in build dir when failed.

[proletarius101]

First you should use git source instead of tar.gz, as it has submodules.

If I

      - type: git
        url: https://github.com/standardnotes/desktop
        tag: v3.5.18

Then it will

FB: Running: git fetch -p --no-recurse-submodules --depth=1 -f origin '+refs/tags/3.5.17:refs/tags/3.5.17'
remote: Enumerating objects: 271, done.
remote: Counting objects: 100% (271/271), done.
remote: Compressing objects: 100% (251/251), done.
remote: Total 271 (delta 10), reused 132 (delta 5), pack-reused 0
Receiving objects: 100% (271/271), 735.50 KiB | 799.00 KiB/s, done.
Resolving deltas: 100% (10/10), done.
From https://github.com/standardnotes/web
 * [new tag]         3.5.17     -> 3.5.17
FB: Running: git rev-parse 687b53263f379466294a320e62204bfcada997cb
FB: Running: git rev-parse --verify --quiet 687b53263f379466294a320e62204bfcada997cb:.gitmodules
FB: Running: git show 687b53263f379466294a320e62204bfcada997cb:.gitmodules
FB: Running: git ls-tree 687b53263f379466294a320e62204bfcada997cb vendor/extensions/extensions-manager
Failed to download sources: module standardnotes: Not a gitlink tree: vendor/extensions/extensions-manager

[proletarius101]

You need a proxy if you are in China, and refresh you proxy tool,dns cache then try more times.

No I'm not. And sorry I pasted the wrong log. Edited

[catsout]

Seems no way to fix, you may need add source manually.

[proletarius101]

Seems no way to fix, you may need add source manually.

That, however, is probably a bug of flatpak-builder. standardnotes repo uses a nested submodule structure, while flatpak builder tries to git ls-tree submodules of the module web in the parent dir, which obvious is incorrect.

[catsout]

That, however, is probably a bug of flatpak-builder. standardnotes repo uses a nested submodule structure, while flatpak builder tries to git ls-tree submodules of the module web in the parent dir, which obvious is incorrect.

Maybe not this, as directly using standardnote/web's git source get the same error.
It's seems that standardnote/web has .gitmodules but deleted the gitlink entries, which cause this error.

[proletarius101]

That, however, is probably a bug of flatpak-builder. standardnotes repo uses a nested submodule structure, while flatpak builder tries to git ls-tree submodules of the module web in the parent dir, which obvious is incorrect.

Maybe not this, as directly using standardnote/web's git source get the same error.
It's seems that standardnote/web has .gitmodules but deleted the gitlink entries, which cause this error.

That's interesting... It doesn't even clone the vendor/extensions/extensions-manager in vendor/extensions/extensions-manager!

[proletarius101]

@catsout Thanks for pointing me to the right direction. standardnotes/app#516 fixes the corrupted submodules for this case.

On the other hand, looking forward to your PR being merged ;)

[proletarius101]

So, there is another case of git-based dependencies not covered.
e.g. https://github.com/bitwarden/jslib/blob/859f317d59189d223072a406bc2d6924e1fb71bc/package-lock.json#L3262, there's a git+ssh source which is valid according to the doc. But the generator raises an error:

AssertionError: git+ssh://git@github.com/duosecurity/duo_web_sdk.git#410a9186cc34663c4913b17d6528067cd3331f1d doesn't match any Git prefix