RFC: Decoupled Configuration and Build Matrix Support

[nrdxp]

Context & Motivation

This proposal outlines a plan for decoupling configuration from derivation in Eos. This represents a significant departure from the current de facto standard in Nix, which typically involves generating all possible package variants directly in raw Nix code.

The goal is to modernize this approach by adopting build matrices, a standard practice in modern CI/CD and build systems, while retaining the formal correctness and cryptographic rigor that Nix provides. By decoupling configuration from the derivation but still tracking the configuration hash cryptographically, we can support structured build requests and dynamic matrix generation without sacrificing reproducibility.

Request Structure

Eos would accept a request structure where configuration is explicitly defined and separated from the package definition. Here is an example of what a request might look like:

{
  "request": [
    {
      "config": {
        "downstream": {
          "<version-rev-or-id>": {
            "copt": [
              [
                "-DSOME_WORKAROUND=1"
              ]
            ],
            "defines": [
              [
                "BROKEN_FEATURE=0"
              ]
            ]
          }
        },
        "inherited": {
          "build": [
            "x86_64-linux"
          ],
          "target": [
            "x86_64-linux",
            "aarch64-linux"
          ]
        },
        "local": {
          "rust": {
            "features": [
              [
                "my-feat",
                "other"
              ],
              [
                "some-other"
              ]
            ]
          }
        }
      },
      "id": "<id>",
      "package": {
        "label": "",
        "version": ""
      },
      "rev": "<version-rev>"
    }
  ]
}

Configuration Types

• Inherited: Global configuration propagated to all downstream dependencies. This should follow a well-known, globally abstracted schema (similar to how build phases are abstracted in Nixpkgs) to ensure all atoms can reliably conform to it. Examples include cross-compilation targets, standard build flags, and configure flags.
• Local: Configuration specific to the current atom (e.g., rust.features).
• Downstream: Overrides for local configuration values of dependencies (e.g., copt, defines for a atom or a specific revision).

Schema Definition via atom.toml

A crucial detail is that the local configuration schema is determined by the atom.toml itself.

• Atoms will specify default values for their local configuration in atom.toml.
• The value and type of these defaults effectively constitute the schema for what can be overridden.
• This ensures a structured understanding of what is configurable without the risk of the schema drifting from the actual declarations in the atom.

Build Matrix Generation

Every configuration value in the request is a list. This structure allows Eos to build a (potentially sparse) matrix of configurations on the fly.

For the request above, the actual builds would receive a matrix similar to this:

inherit.build	inherit.target	local.rust.features	downstream."".copt	downstream."".defines
x86_64-linux	x86_64-linux	my-feat, other	-DSOME_WORKAROUND=1	BROKEN_FEATURE=0
x86_64-linux	x86_64-linux	some-other	-DSOME_WORKAROUND=1	BROKEN_FEATURE=0
x86_64-linux	aarch64-linux	my-feat, other	-DSOME_WORKAROUND=1	BROKEN_FEATURE=0
x86_64-linux	aarch64-linux	some-other	-DSOME_WORKAROUND=1	BROKEN_FEATURE=0

Usage Examples

These flags could be passed via the Eka CLI or structured in a TOML file for CI jobs to ensure all relevant variants are built:

[[request]]
id = "<id>"
rev = "<version-rev>"
# build matrix variations (global propagated to all dependencies)
config.inherited = { build = ["x86_64-linux"], target = ["x86_64-linux", "aarch64-linux"] }
# more matrix variants, local to the given atom
config.local = { rust.features = [["my-feat", "other"], ["some-other"]] }
config.downstream."<version-rev-or-id>" = { copt = [["-DSOME_WORKAROUND=1"]], defines = [["BROKEN_FEATURE=0"]] }
package = { label = "", version = "" }

Evaluation Logic

Each actual evaluation will have a concrete configuration passed in, like so:

{
  "config": {
    "downstream": {
      "<version-rev-or-id>": {
        "copt": [
          "-DSOME_WORKAROUND=1"
        ],
        "defines": [
          "BROKEN_FEATURE=0"
        ]
      }
    },
    "inherited": {
      "build": "x86_64-linux",
      "target": "aarch64-linux"
    },
    "local": {
      "rust": {
        "features": [
          "my-feat",
          "other"
        ]
      }
    }
  },
  "id": "<id>",
  "package": {
    "label": "",
    "version": ""
  },
  "rev": "<version-rev>"
}

This JSON would be normalized and hashed to track previously recorded evaluations.

• Caching: The atom cache (using Git as a global blob store) can use symbolic refs.
• Ref Structure: A ref of (config-hash + rev) can be a symref pointing to another ref of the derivation hash, which in turn points to the actual atom ref (label + version).
• Resolution: When hitting the derivation ref, Eos can check the local /nix/store for availability, skipping evaluation if the result is already present.

Future Work & Notes

• This is a proposal for the eos daemon service, which will handle scheduling and responding to these requests.
• Config Only Atoms: Future work may include "config only atoms" that preconfigure a host of matrix configurations. These could be pulled in as "config dependencies" by upstream consumers to share configuration without redeclaring it.