Make _get_program_from_buffer work for bundled programs take 2

[Erik-Lundell]

This is a re-upload of #14435, with variables moved inside #ifdef to not get unused variables when building without devtools.

Add some cmake to only do this if executorch is built with bundleio.

codegen/tools subdirectory include needs to be moved in top-level CmakeLists.txt
to have access to the bundled_program target.

Follow-up patch to enable the fix in arm backend.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14503

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 2 Cancelled Jobs, 3 Unrelated Failures

As of commit 390840c with merge base df5bfd5 ():

NEW FAILURES - The following jobs have failed:

• pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t cc9f3dc2deb6122c7508e72355275deaafb51be858b83b140c216d07c72674b4 /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t 331924cce45ebe60317c234cf9d35f717e35c0b6dc08331b36071fbdef77413a /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t 49e79ab85973013eef9cd47b196904060395606bd2633d13f39e9d9b377a9a68 /exec failed with exit code 1
• pull / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 0bf9574e6fd35afdbb131c41f7a2f9da7e24c7a327de6cff32549479ec221a88 /exec failed with exit code 1
• trunk / test-arm-ootb-linux / linux-job (gh)
  RuntimeError: Command docker exec -t e48517c57963ce69420160c6a8232e5fa4a7a9d9b6200556f7dc1b4f9a9a7129 /exec failed with exit code 1

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• pull / test-binary-size-linux-gcc / linux-job (gh)
  ##[error]The operation was canceled.
• pull / test-setup-linux-gcc / linux-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  exir/tests/test_quant_fusion_pass.py::TestQuantFusionPass::test_embedding_torchao
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  exir/tests/test_quant_fusion_pass.py::TestQuantFusionPass::test_embedding_torchao
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  exir/tests/test_quant_fusion_pass.py::TestQuantFusionPass::test_embedding_torchao

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

Merge should wait until #14488 to make sure Meta's internal build is not broken

[mergennachin]

Should you check EXECUTORCH_BUILD_DEVTOOLS?

Something like

if (EXECUTORCH_BUILD_DEVTOOLS and TARGET bundled_program)
   ... 

[Erik-Lundell]

I went with this since it is done that way on L754 in top-level CMakeLists.txt. Is there a benefit in additionally checking the flag or just to be on the safe side?

[mergennachin]

You also wanna change the targets.bzl file too

I added a CI to build buck target for selective_build

#14499

For future reference, here's how you can download buck:

python tools/cmake/resolve_buck.py --cache_dir==/tmp/
# it will download and output a buck executable file within /tmp/ file

Once you have the buck file, you can do something like this:

/tmp/buck2 build //codegen/tools/... 
# build all targets within //codegen/tools/...

/tmp/buck2 build codegen/tools:selective_build
# build all targets

[Erik-Lundell]

I had a look and the bundled_program seems to have multiple targets runtime and runtime_aten, depending on some get_aten_mode_options(). I need some assistance to figure this out. From what I can tell, if I add the different targets to selective_build that target is also split which means that targets depending on it will also split and so on... That doesn't sound right.

[mergennachin]

@Erik-Lundell -- no need to bifurcate on aten vs not. You can add dependency on "//executorch/devtools/bundled_program:runtime

Historically, runtime_aten is mainly for internal customers when portable ops coverage was low. That's not the case, so you should be able to just depend on "//executorch/devtools/bundled_program:runtime directly

[facebook-github-bot]

@mergennachin has imported this pull request. If you are a Meta employee, you can view this in D83148788.

[digantdesai]

Looks good with internal CI as well. Stamping..