nix flake tracking issue

[nothingmuch]

Tracking issue for various flake related potential improvements.

• cargo audit
  • update direct dependencies to address existing flags
  • enforce with flake check, see also Cargo-minimal.lock and Cargo-recent.lock have the same contents. Is this right? #454

Details

# another flake input is required for the advisory db
inputs.advisory-db = {
  url = "github:rustsec/advisory-db";
  flake = false;
}

# and new flake checks
payjoin-workspace-audit-minimal = craneLib.cargoAudit {
  inherit src advisory-db;
  cargoLock = ./Cargo-minimal.lock;
};
payjoin-workspace-audit-recent = craneLib.cargoAudit {
  inherit src advisory-db;
  cargoLock = ./Cargo-recent.lock;
};

• cargo deny
  • add deny.toml with approved licenses
  • enforce with flake check:

Details

payjoin-workspace-deny-minimal = craneLib.cargoDeny {
  inherit src;
  cargoLock = ./Cargo-minimal.lock;
};
payjoin-workspace-deny-recent = craneLib.cargoDeny {
  inherit src;
  cargoLock = ./Cargo-recent.lock;
};

• toml formatting
  • format existing files
  • enforce with flake check

Details

payjoin-workspace-toml-fmt = craneLib.taploFmt {
  src = pkgs.lib.sources.sourceFilesBySuffices src [ ".toml" ];
};

• cargo machete or cargo udeps
  • remove unused dependencies
  • add flake check similar to this
• CI
  • non build/test related flake checks on push
    • https://github.com/DeterminateSystems/nix-installer-action
    • nix flake check -L .#{nix-fmt-check,shfmt,shellcheck}
  • periodic flake update (update nightly rust in devshells)
    • https://github.com/DeterminateSystems/nix-installer-action
    • https://github.com/DeterminateSystems/magic-nix-cache-action
    • https://github.com/DeterminateSystems/update-flake-lock
    • https://github.com/DeterminateSystems/flake-checker
    • full nix flake check -L, ensures project still compiles and runs unit tests but not integration tests for now
• integration tests
  • add pkgs.bitcoind to environments (devshell, testing)
  • add pkgs.redis to environment, and allow running it without docker (bypassing test containers)
  • run full nix flake check in CI
  • update contributing section of README to document nix flake check usage, caveats
• export test artifacts
  • clippy https://github.com/carol-computer/carol/blob/09c8ff24a732a84c4fc433c091681837a2545446/flake.nix#L276-L285
  • tests, coverage? https://nexte.st/docs/configuration/#profiles
  • docs?
• ensure payjoin-cli README examples work (https://github.com/carol-computer/carol/blob/09c8ff24a732a84c4fc433c091681837a2545446/flake.nix#L230-L268)

[nothingmuch]

@DanGould input re priorities would be appreciated. these seem more important:

1. making sure nightly rust is reasonably up to date and working seems valuable
2. work around the redis issue in order to make nix flake check more useful
3. fixing cargo audit or at least triaging its complaints, there's some unmaintained stuff in our dependency chain right now:

error: 1 vulnerability found! warning: 6 allowed warnings found

Crate:     idna
Version:   0.5.0
Title:     `idna` accepts Punycode labels that do not produce any non-ASCII when decoded
Date:      2024-12-09
ID:        RUSTSEC-2024-0421
URL:       https://rustsec.org/advisories/RUSTSEC-2024-0421
Solution:  Upgrade to >=1.0.0
Dependency tree:
idna 0.5.0
└── url 2.5.0
    ├── tungstenite 0.21.0
    │   ├── tokio-tungstenite 0.21.0
    │   │   ├── ohttp-relay 0.0.9
    │   │   │   ├── payjoin-cli 0.0.9-alpha
    │   │   │   └── payjoin 0.22.0
    │   │   │       └── payjoin-cli 0.0.9-alpha
    │   │   └── hyper-tungstenite 0.13.0
    │   │       └── ohttp-relay 0.0.9
    │   └── hyper-tungstenite 0.13.0
    ├── reqwest 0.12.4
    │   ├── payjoin-cli 0.0.9-alpha
    │   └── payjoin 0.22.0
    ├── redis 0.23.3
    │   └── payjoin-directory 0.0.1
    │       ├── payjoin-cli 0.0.9-alpha
    │       └── payjoin 0.22.0
    ├── payjoin-cli 0.0.9-alpha
    ├── payjoin 0.22.0
    └── bhttp 0.5.1
        ├── payjoin-directory 0.0.1
        └── payjoin 0.22.0

Crate:     atty
Version:   0.2.14
Warning:   unmaintained
Title:     `atty` is unmaintained
Date:      2024-09-25
ID:        RUSTSEC-2024-0375
URL:       https://rustsec.org/advisories/RUSTSEC-2024-0375
Dependency tree:
atty 0.2.14
└── env_logger 0.9.3
    └── payjoin-cli 0.0.9-alpha

Crate:     instant
Version:   0.1.13
Warning:   unmaintained
Title:     `instant` is unmaintained
Date:      2024-09-01
ID:        RUSTSEC-2024-0384
URL:       https://rustsec.org/advisories/RUSTSEC-2024-0384
Dependency tree:
instant 0.1.13
├── parking_lot_core 0.8.6
│   └── parking_lot 0.11.2
│       └── sled 0.34.7
│           └── payjoin-cli 0.0.9-alpha
└── parking_lot 0.11.2

Crate:     proc-macro-error
Version:   1.0.4
Warning:   unmaintained
Title:     proc-macro-error is unmaintained
Date:      2024-09-01
ID:        RUSTSEC-2024-0370
URL:       https://rustsec.org/advisories/RUSTSEC-2024-0370
Dependency tree:
proc-macro-error 1.0.4
└── clap_derive 4.0.21
    └── clap 4.0.32
        └── payjoin-cli 0.0.9-alpha

Crate:     yaml-rust
Version:   0.4.5
Warning:   unmaintained
Title:     yaml-rust is unmaintained.
Date:      2024-03-20
ID:        RUSTSEC-2024-0320
URL:       https://rustsec.org/advisories/RUSTSEC-2024-0320
Dependency tree:
yaml-rust 0.4.5
└── config 0.13.4
    └── payjoin-cli 0.0.9-alpha

Crate:     atty
Version:   0.2.14
Warning:   unsound
Title:     Potential unaligned read
Date:      2021-07-04
ID:        RUSTSEC-2021-0145
URL:       https://rustsec.org/advisories/RUSTSEC-2021-0145

Crate:     futures-util
Version:   0.3.30
Warning:   yanked
Dependency tree:
futures-util 0.3.30
├── tower 0.4.13
│   └── hyper-util 0.1.7
│       ├── reqwest 0.12.4
│       │   ├── payjoin-cli 0.0.9-alpha
│       │   └── payjoin 0.22.0
│       │       └── payjoin-cli 0.0.9-alpha
│       ├── payjoin-directory 0.0.1
│       │   ├── payjoin-cli 0.0.9-alpha
│       │   └── payjoin 0.22.0
│       ├── payjoin-cli 0.0.9-alpha
│       ├── ohttp-relay 0.0.9
│       │   ├── payjoin-cli 0.0.9-alpha
│       │   └── payjoin 0.22.0
│       ├── hyper-tungstenite 0.13.0
│       │   └── ohttp-relay 0.0.9
│       └── hyper-rustls 0.26.0
│           ├── reqwest 0.12.4
│           ├── payjoin-directory 0.0.1
│           ├── payjoin-cli 0.0.9-alpha
│           └── ohttp-relay 0.0.9
├── tokio-tungstenite 0.21.0
│   ├── ohttp-relay 0.0.9
│   └── hyper-tungstenite 0.13.0
├── reqwest 0.12.4
├── redis 0.23.3
│   └── payjoin-directory 0.0.1
├── hyper-util 0.1.7
├── hyper-rustls 0.26.0
├── hyper 1.4.1
│   ├── reqwest 0.12.4
│   ├── payjoin-directory 0.0.1
│   ├── payjoin-cli 0.0.9-alpha
│   ├── ohttp-relay 0.0.9
│   ├── hyper-util 0.1.7
│   ├── hyper-tungstenite 0.13.0
│   └── hyper-rustls 0.26.0
├── http-body-util 0.1.2
│   ├── reqwest 0.12.4
│   ├── payjoin-directory 0.0.1
│   ├── payjoin-cli 0.0.9-alpha
│   ├── ohttp-relay 0.0.9
│   └── hyper-tungstenite 0.13.0
├── futures-executor 0.3.30
│   └── futures 0.3.30
│       ├── testcontainers 0.15.0
│       │   ├── testcontainers-modules 0.1.4
│       │   │   ├── payjoin-cli 0.0.9-alpha
│       │   │   └── payjoin 0.22.0
│       │   ├── payjoin-cli 0.0.9-alpha
│       │   └── payjoin 0.22.0
│       ├── payjoin-directory 0.0.1
│       └── ohttp-relay 0.0.9
└── futures 0.3.30

error: 1 vulnerability found!
warning: 6 allowed warnings found

[DanGould]

Another thing I'd really like with this is environments for payjoin-ffi languages where we can just call payjoin-ffi/python/contrib/test.sh like we do with all of the other targets. This is a pain point for me in testing at the moment since python has its own test setup unlike the main rust workspace.

@thebrandonlucas

[thebrandonlucas]

@DanGould if I understand this ask correctly, we want to utilize nix (as an additional option in the top-level flake.nix or as a new flake.nix in payjoin-ffi/python (or other language)) to spin up an environment for the target language that would allow you to run that language's tests via contrib/<language>/test.sh?

In which case the ask is twofold: create a nix shell.nix/flake.nix in the payjoin-ffi/<language> which will spin up the environment that would allow us to run tests, and then create a payjoin-ffi/<language>/contrib/test.sh which will actually run the test suites (for python, this would be the test_payjoin_integration_test.py and test_payjoin_unit_test.py). I do not currently see any contrib/test.sh in there.

My bias would really be to create a flake for each language which would allow us to simply run all the lints/tests by running nix flake check in a single command rather than having to enter the environment and then run test.sh manually. If you did just want to enter the environment to run them manually, that would still be possible with nix develop

[DanGould]

@thebrandonlucas I do want a nix flake that provides the environment for us to compile, release, and run language bindings and their tests. Ideally everything was in a single top level flake I could nix develop into. Yes, you'd need to create both the flake (or add to what we have) and a test script. If this can be done with multiple flakes that compose, each nix flake checkable that'd be great. You've got more nix experience than I do at this point so again, I'm going to trust your judgement seeing as we both agree afaict that having some reproducible tooling here would simplify testing for everyone.

[DanGould]

@nothingmuch reinforced to me that indeed each language can have its own flake that composes at the top level and recommended https://github.com/the-nix-way/dev-templates as a reference.

[DanGould]

At this point nix integration tests seem to be the priority. We're struggling with dart/redis/docker combo and eliminating docker would be a great help in that it would let us run dart tests in CI and across environments with sufficient reliability, presumably.

[nothingmuch]

the nix flake check for running cargo nextest also runs with --all-features, so we probably want to split those up into multiple jobs for the main feature combinations we run