Support for `pre_compile` and `post_compile` steps

[ipmb]

Just tested out v0.1.0 and noticed this was missing. It looks like you have some other issues to get it to parity with the legacy buildpack, so just dropping this one in too.

Thanks for your work on this!

[edmorley]

Thank you! Yeah this is on the list of things to consider - though it's possible I'll be dropping support for this in favour of people using the built-in inline buildpack feature instead, since it's not ideal for each language to reimplement custom run support. (For languages where there are built-in hooks, it's perhaps still worth supporting those, but for Python post_compile is proprietary so doesn't seem worth preserving in the CNB.)

I'll leave this open to track that decision, and if the feature is dropped we'll need to add warning (or error) messages to ease the transition regardless.

[luzfcb]

I haven't tried this new way of deployment yet, but I would like to leave a comment on the way I use post_compile today.
My use case for post_compile is as follows:

I have an e-commerce platform made with Django + Frontend application in React. The deployment process has integration with Sentry releases.

In Heroku Classic, you do not have access to GIT and repository, just an environment variable that informs which git commit hash of a particular deploy flow has started.
You also do not have access to information when using pipelines and use functionality to promote to Production a Heroku Slug running on Staging

The application has 2 independent react applications (one uses old things and another uses new and incompatible things, and we are gradually migrating to the new application)

We use pre_compile for:
1 - Perform compilation tasks for the 2 react applications, so that everything is prepared for when python/django compilation tasks are performed
2 - Upload to Sentry releases of the Source Maps JS files generated in the previous step
3 - Since Heroku Classic has no direct access to the Git Repository, Sentry-Cli is not able to automatically get all the information needed to create a new release in Sentry Releases, so I must do so manually by running a custom script to consult the Sentry API to find out which one was the hash of last previous deployment made with successful status (That is, a finalized release) and then generate the range of commits to which this deploy possibly covers and manually starts the creation of a new release on Sentry Releases

post_compile:

1 - We perform Django's Collectstatic
2 - If all previous steps are executed sucessfully, we execute the Finalize release step of Sentry Releases to confirm that the application was successfully implanted. https://docs.sentry.io/product/cli/releases/#finalizing-releaseseses

[edmorley]

For CNBs, the standardised way of running custom commands before/after buildpacks is now using the new built-in inline buildpacks feature, which is documented here:
https://buildpacks.io/docs/app-developer-guide/using-inline-buildpacks/

For language ecosystems where there is a standardised convention (for that language) way of running commands, I think it makes sense for the buildpacks for that language to keep supporting those conventions (for example, the Node.js buildpack supporting the package.json scripts feature for things like the build command etc).

However, for language ecosystems like Python where there is no such convention, I don't think it makes sense to preserve the classic Python buildpack's proprietary bin/{pre,post}_compile steps, given there is a built in feature for custom commands in the CNB spec (inline buildpacks).

As such, I think I may not end up adding support for this in the Python CNB. Though I may end up adding some explicit error handling to ease the transition. (Which could for example print an example inline buildpack config to the build log, which users can then copy directly into their project.toml.)

[ipmb]

I understand the motivation here, but project.toml feels like a big regression in DX/ergonomics.

Previously I could do something like this:

pack build myimage -b heroku/nodejs,heroku/python -B heroku/buildpacks:20

If I'm understanding the inline buildpacks, I now need to add a project.toml like this:

[_]
schema-version = "0.2"
id = "io.buildpacks.my-app"

[[io.buildpacks.group]]
id = "heroku/nodejs"
version = "1.1.2"

[[io.buildpacks.group]]
id = "heroku/python"
version = "0.4.0"

[[io.buildpacks.group]]
id = "me/post-compile"

  [io.buildpacks.group.script]
  api = "0.9"
  inline = "echo hello"

and then I can run:

pack build myimage -B heroku/builder:22

Versions seem required in project.toml so you lose auto-updating. It also requires more knowledge of buildpack internals (schema_version, api, io.buildpacks...).

Maybe this is outside of Heroku's scope, but a thought I had was that a separate "run-script" buildpack could be published which executes a script from a predefined location (or location provided by env var)?

[edmorley]

The buildpack version in project.toml is an optional field, and defaults to "latest", so auto-updating of buildpack versions won't be affected. See:
https://buildpacks.io/docs/reference/config/project-descriptor/#iobuildpacks-_table-optional_

I agree that users may get the syntax of the project.toml file incorrect, however, we can and should document it clearly + have clear validation error messages (either upstream in the CNB lifecycle, or else in Pack CLI + the Heroku next gen build system).

Ref having to create a project.toml - in your example you would already have had to create such a file, since that app uses both Python and Node.js, and the CNB auto-detection would have only picked one language. (Just like currently Heroku only auto-detects a single language when no buildpacks are set, and for anything else you have to set explicit buildpacks using heroku buildpacks:set.)

Whilst the project.toml file won't be mandatory, it will be needed for any app that:

• uses more than one official language buildpack (as above)
• uses a third party buildpack
• wants to pin to a non-latest version of an official buildpack
• wants to use a non-default Heroku stack (or more correctly, CNB builder), or wants to control when the stack is updated rather than relying on the new Heroku default builder concept

In addition, app.json may eventually end up being merged into project.toml, meaning that a project.toml file may be needed for the Heroku CI / Review App use-cases too.

As such, I think having a project.toml will be fairly common for anything but beginner apps. And beginner apps don't tend to use bin/{pre,post}_compile.

In general though, I much prefer having buildpacks/stacks be configured via code rather than heroku buildpacks:set / heroku stack:set, since implicit differing state between apps is a regular source of user confusion ("why does my staging app work but my production app not, I'm git pushing the same source to both" etc).

[edmorley]

a thought I had was that a separate "run-script" buildpack could be published which executes a script from a predefined location

Yeah this is another viable solution too.

Though perhaps a third option would be to ask for a simpler project.toml syntax for "run commands" that don't require the full verbosity of the current "inline buildpack" concept?
https://github.com/buildpacks/rfcs#proposal

[ipmb]

The buildpack version in project.toml is an optional field, and defaults to "latest", so auto-updating of buildpack versions won't be affected.

Using the latest pack, I get this error if I remove the versions from the heroku buildpack groups:

$ pack build myimg -B heroku/builder:22
22: Pulling from heroku/builder
Digest: sha256:c9f45adaaf3f91eb2f126b29c8e947f1375bae560a725511393b6a197cfe765d
Status: Image is up to date for heroku/builder:22
22-cnb: Pulling from heroku/heroku
Digest: sha256:0c422f13416d589a199e3fbf9fa17c1a75b1a120fc60425cd6a20216c432f026
Status: Image is up to date for heroku/heroku:22-cnb
ERROR: failed to build: Invalid buildpack defined in project descriptor

If I use "latest" I get:

$ pack build myimg -B heroku/builder:22
22: Pulling from heroku/builder
Digest: sha256:c9f45adaaf3f91eb2f126b29c8e947f1375bae560a725511393b6a197cfe765d
Status: Image is up to date for heroku/builder:22
22-cnb: Pulling from heroku/heroku
Digest: sha256:0c422f13416d589a199e3fbf9fa17c1a75b1a120fc60425cd6a20216c432f026
Status: Image is up to date for heroku/heroku:22-cnb
ERROR: failed to build: downloading buildpack: error reading heroku/nodejs@latest: invalid locator: InvalidLocator

What am I missing?

[ipmb]

Thanks for the thoughtful write-up @edmorley! I see where you're coming from and the value in following a standard. If good examples are provided, the project.toml is bearable, but heroku buildpacks:set feels far more user friendly 🙂

[edmorley]

What am I missing?

Oh strange! I know there are some heuristics around interpreting a plain URI identifier (ie: to work out if it's a Docker Hub reference, or say a CNB registry reference; xref https://buildpacks.io/docs/app-developer-guide/specify-buildpacks/#uri-examples), perhaps there is a bug around that and the default of latest? I personally haven't tried defining buildpacks in project.toml yet (we're still very early stages in the overall CNB / CNB tooling journey), so maybe the best place to ask for now might be https://github.com/buildpacks/pack/issues or https://slack.cncf.io/ (in one of the #buildpacks / #buildpacks-* channels).

Edit: This bug was fixed upstream by buildpacks/pack#1873.

but heroku buildpacks:set feels far more user friendly 🙂

I agree it is seemingly more friendly at first glance, but it actually does cause quite a few issues that likely do not surface for experienced/diligent users (such as yourself :-)) but often trip beginners up - such as:

• the classic footgun of "buildpacks:set" clobbering existing buildpacks
• users not realising buildpacks have an order, or knowing about --index to buildpacks:set or buildpacks:add
• users updating the buildpacks in app.json after an app was first created (rather than running buildpacks:{add,set}), and not knowing why the app doesn't run the new buildpacks. (Particularly for Review Apps, since the buildpacks in app.json are used for the first build, but not for subsequent)
• different apps in their account running the same codebase but showing different behaviour and users not knowing why (either because of them having completely different buildpacks, or even more subtle cases of one app being pinned to an older version of an official buildpack compared to another)

In addition, with CNBs we finally have a much better story for running a build locally that matches production in the form of pack build etc. However, given that heroku buildpacks:set stores state remotely in Heroku's API/database for a specific app, how would pack build know what buildpacks to run, unless that's defined in the codebase?

[ipmb]

In case anyone else comes across this, a uri is required, not an id for it to work without a version:

[[io.buildpacks.group]]
uri = "urn:cnb:builder:heroku/nodejs"

[edmorley]

In case anyone else comes across this, a uri is required, not an id for it to work without a version:

@ipmb That seems like a bug - would you mind reporting it upstream? https://github.com/buildpacks/pack/issues

[ipmb]

✅ buildpacks/pack#1862

[edmorley]

Thank you!