Use custom build comand for editable install of the project

[vigneshmanick]

Problem description

We have a python c++ project and there are other commands that need to be run before running python -m pip install -e . , setting environment variables(CMAKE_ARGS) , preparing necessary data etc.

For this purpose we have created an install.py which works cross platform. It would awesome if a custom command could be specified for the pyproject.toml editable dependency that has been implemented in #1084

example

[pypi-dependencies]
minimal-project = { path = ".", editable = true, build_cmd=["python install.py develop"]}

[olivier-lacroix]

@vigneshmanick what about using a pyproject.toml manifest and a build-backend allowing you to do this?

[vigneshmanick]

am already using a build backend but still some variables/steps need to be executed so that the build backened understands it.

Example prod and development build

development build
curl <test file> && CMAKE_ARGS="-DEXPERIMENTAL_FEATURE=on" pip install -e .

prod build
curl <prod file> && CMAKE_ARGS="-DEXPERIMENTAL_FEATURE=off" pip install -e .

Generally builds are not as simple as pip install . there is always some customization that needs to be done both at the build backened and preparation stage.

Currenly i have removed the pypi-dependencies section and instead use an install task to achieve the editable install of the package.

[tdejager]

Maybe you could do this in an activation script of the environment? Wdyt @ruben-arts

[vigneshmanick]

activation script means writing one for each platform which is an overhead and also it seperates a build step out of the actual build part. These are not variables that are acutally required in the environment but only for building.

Let me give more detail for the scenario that i have

1. we have different config files for different build configurations which result in different cmake args
2. the build step can be changed by specifying which config file to use/ and also the build mode (e.g python install.py prod, python install.py dev). The config files are read by configparser module and the necessary variables are passed to cmakelists.txt
3. The build wheel also requires data files that are not part of the repository but are downloaded from multiple other repositories
4. All of the above are handled via an install.py which handles these tasks before build starts and then builds the project (editable for development, wheel for prod which is then used as artifact for further pipelines)

Allowing a custom install command will solve all of the above instead of having to resort to splitting the processes. This also is more maintainable since the entire build logic is in one place and not split.

[olivier-lacroix]

Maybe I am missing something, but I thought some build-backends would allow you to do this?

For instance, hatchling has a plug-in concept, that allows you to modify a fair bit of the build process.

scikit-build-core may be another build backend to consider, focused on compiled use cases.

[vigneshmanick]

I am using both scikit-build-core and scikit-build . The above steps still apply.

[ruben-arts]

Maybe you could do this in an activation script of the environment? Wdyt @ruben-arts

@vigneshmanick This pr could possibly help you in the short term, would you like it? I'm holding it back because if the issue described in the PR but it might already help you. #1156

I'm not sure if custom build commands would open a can of worms for us, but I can see the use-case. We should still take a look at the "build environment" for pypi packages so the environment variables would most-likely also be a part of that.
#1130 #1110

[vigneshmanick]

Thanks for the response, i checked the #1156 and is definitely very helpful for tasks within the environment but am afraid won't change much to the situation in this issue.

A custom build environment as mentioned in #1110 would definitely make the process more flexible since this step isoloated.

For my understanding, the dependencies (pypi /conda) need to be solved for all the environments/features/platforms so that you can ensure that it's valid and this is why you are hesitant to add a custom command?

From my point of view, if i as a user have resorted to a custom command, then the entire responsiblity of ensuring that the build works lies on me. In this case, i would expect that pixi just ignores solving this custom dependency and solves the others and once complete the custom command is run at the end as is.

[vigneshmanick]

Also to add, i am a bit hesitant to split the build process since the current process works the same if i have a venv , conda or within pixi. Splitting the process will result in more maintenance since the build has to work also when no pixi is available.

[tdejager]

I'm a bit hesitant to add it because it would partially circumvent the entire build-backend in the pyproject.toml and I feel that build backends like meson or anything in use by the numerical python world would have a use-case similar to yours. I mean you can do anything you want with a setup.py e.g. But it could be that I'm missing something :)

[tdejager]

After some more investigation:

1. For the custom build steps. E.g. would something like this not be an option for the custom steps? https://github.com/rmorshea/hatch-build-scripts
2. For the cmake commands you could create a seperate build env with activation based on platform, this is forwarded to uv build. If this could be made more ergonomic we can look into it :)

Also in addition to 2), like @olivier-lacroix mentioned: https://github.com/scikit-build/scikit-build-core?tab=readme-ov-file also has support for custom cmake flags (even per platform I think)

This would also alleviate your concerns regarding the split between conda and wheel builds.

[vigneshmanick]

hatch cannot be used for my use case

Yes CMAKE_ARGS can be defined but i want to modify the CMAKE_ARGS before building dynamically

in addition, customizing the build is one step but i need both build customization and src customization (i need to gather extra data files from other sources which are also part of the wheel based on other parameters) which is not possible(most probably never will) in any of the current build backends.

Currently my build works if i remove

[pypi-dependencies]
minimal-project = { path = ".", editable = true,}

and just add

[tasks]
install = "python install.py develop"

pixi install && pixi run install

If you feel that this is not a use case for pypi-dependencies/hesitant to introduce , that's perfectly fine. I just want to bring to your attentiont that such a feature would improve the QOL /extend the reach in using pixi since all the plumbing is already there :) . As you had already mentioned, this is not an uncommon scenario. Users will always find a way to stretch any tool to it's limits (maybe even repurpose it to cases for which it was not foreseen)

[tdejager]

What's the biggest benefit of this instead of doing the work in a setup.py and using setuptools?. I know the SDL backend on pypi is doing something like this to download the .so files.

[vigneshmanick]

Biggest benefit is that i have a install script that works cross platform / across python platforms (pixi, conda , venv) and also maintain a seperation between what modification is required for the project (this is within pyproject.toml ) and what's required for the build clearly seperated.

setup.py is outdated and the build systems are moving /already moved to pyproject.toml only based build, some build systems will not recognize the setup.py and expect the project data to be in pyprojec.toml / env variables.

In my case the entire setup.py contents has been removed replaced with pyproject.toml + install script.