arch/xtensa/esp32[-s2|-s3]: Modify the method of downloading the repository #17236

TangMeng12 · 2025-10-23T10:49:48Z

Directly downloading the Git repository is inconvenient for local debugging.

Directly downloading the Git repository is inconvenient for local debugging.
This will allow to automatically download external packages from the Internet.
If not set, the repo need to be download will need to provide them manually,
otherwise an error will occur and the build will be aborted.

Add USE_NXTMPDIR_ESP_REPO_DIRECTLY, with this we can use
USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make which can directly use esp-hal-3rdparty
under nxtmpdir without CHECK_COMMITSHA, reset, checkout and update.

Note: Please adhere to Contributing Guidelines.

Summary

Add USE_NXTMPDIR_ESP_REPO_DIRECTLY, with this we can use
USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make which can directly use esp-hal-3rdparty
under nxtmpdir without CHECK_COMMITSHA, reset, checkout and update.

nuttxworkspace
|
|- nuttx
|- apps
|- nxtmpdir

USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make

ARCH
arch/xtensa/src/esp32/Make.defs
arch/xtensa/src/esp32s2/Make.defs
arch/xtensa/src/esp32s3/Make.defs
arch/risc-v/src/common/espressif/Make.defs
arch/risc-v/src/esp32c3-legacy/Make.defs

Impact

There should be no impact.

Testing

CI

simbit18 · 2025-10-23T14:26:00Z

@TangMeng12 Please look here
#13301

The idea behind this logic was to reduce the time needed for the jobs.

With the change you want to make, build times on CI will be longer than before.

hartmannathan · 2025-10-26T18:54:30Z

@TangMeng12 Please look here #13301

The idea behind this logic was to reduce the time needed for the jobs.

With the change you want to make, build times on CI will be longer than before.

Throughout NuttX, and mostly in the nuttx-apps repo, there are various optional components that might be downloaded.

I think we need to discuss (on the mailing list) and come up with a systematic approach that balances everyone's needs in a suitable way.

Some of those needs are:

In Kconfig, there should be one permission config "Allow downloads", help text: "Allow the NuttX build system to automatically download external packages from the Internet." Suggested name: CONFIG_ALLOW_DOWNLOADS. In the build system, downloading will occur through one central function that may call wget, curl, git, or other commands as needed, and that function will return an error if CONFIG_ALLOW_DOWNLOADS is not configured, aborting the build. This is important because some developers will want to ensure there are no automatic downloads. If external packages are needed, those developers will need to provide them manually.
Some developers may allow downloads in general, but will prefer to download some individual package manually. This allows developers to add their own customizations, choose alternate versions, alternate branches, etc. ("Out of tree downloads.") To support this, each package should have its own fine-grained permission Kconfig to enable or disable downloading that package from the internet.
Some developers will prefer to store external packages at custom path on their workstation. To support this, each package needs a Kconfig to provide the custom path on the developer's workstation. This can be combined with the previous item.
Some developers will prefer to allow downloads, but customize the URLs. This way, they can control which server the download will come from. The server could be a local server on their own intranet or perhaps a custom fork on the internet. To support this, each package needs a Kconfig to specify the URL.
The CI system needs to utilize its resources in a smart way, as pointed out by @simbit18 above. So we need a Kconfig to select between downloading the release tarballs or cloning the repository.

Rather than maintaining a hodgepodge of different approaches, a systematic approach will make it easier to add new external dependencies, maintain existing ones, and handle licensing correctly.

Laczen · 2025-10-27T06:10:50Z

@TangMeng12 Please look here #13301

The idea behind this logic was to reduce the time needed for the jobs.

With the change you want to make, build times on CI will be longer than before.

@simbit18, can you add a line to the documentation on how to use the idea proposed in #13301?

TangMeng12 · 2025-10-27T08:36:47Z

@TangMeng12 Please look here #13301
The idea behind this logic was to reduce the time needed for the jobs.
With the change you want to make, build times on CI will be longer than before.

Throughout NuttX, and mostly in the nuttx-apps repo, there are various optional components that might be downloaded.

I think we need to discuss (on the mailing list) and come up with a systematic approach that balances everyone's needs in a suitable way.

Some of those needs are:

In Kconfig, there should be one permission config "Allow downloads", help text: "Allow the NuttX build system to automatically download external packages from the Internet." Suggested name: CONFIG_ALLOW_DOWNLOADS. In the build system, downloading will occur through one central function that may call wget, curl, git, or other commands as needed, and that function will return an error if CONFIG_ALLOW_DOWNLOADS is not configured, aborting the build. This is important because some developers will want to ensure there are no automatic downloads. If external packages are needed, those developers will need to provide them manually.

Some developers may allow downloads in general, but will prefer to download some individual package manually. This allows developers to add their own customizations, choose alternate versions, alternate branches, etc. ("Out of tree downloads.") To support this, each package should have its own fine-grained permission Kconfig to enable or disable downloading that package from the internet.

Some developers will prefer to store external packages at custom path on their workstation. To support this, each package needs a Kconfig to provide the custom path on the developer's workstation. This can be combined with the previous item.

Some developers will prefer to allow downloads, but customize the URLs. This way, they can control which server the download will come from. The server could be a local server on their own intranet or perhaps a custom fork on the internet. To support this, each package needs a Kconfig to specify the URL.

The CI system needs to utilize its resources in a smart way, as pointed out by @simbit18 above. So we need a Kconfig to select between downloading the release tarballs or cloning the repository.

Rather than maintaining a hodgepodge of different approaches, a systematic approach will make it easier to add new external dependencies, maintain existing ones, and handle licensing correctly.

I have updated the commit based on your suggestions. In the commit, the default value of CONFIG_ALLOW_DOWNLOADS is y, which has no impact on the current logic. When developers want to perform local debugging, they can choose not to set CONFIG_ALLOW_DOWNLOADS, and in this case they can configure the repo locally.

simbit18 · 2025-10-27T13:57:25Z

@TangMeng12 , allow me to make a few observations.

As already mentioned #13301, the current system was designed to speed up and reduce errors during the cloning/download step (also possible with Curl downloads if desired) on our CI.

This system on CI makes sense when the packages need to be used in multiple configurations (currently only for Arch Expressif )

jobs Linux (risc-v-04), Linux (xtensa-01), Linux (xtensa-02)

With this simple check

ifneq ($(wildcard $(NXTMPDIR)/.*),)

We can insert any package (if necessary) from the Nuttx or Apps repository into the storage directory.

Storage is enabled with

-S adds the nxtmpdir folder for third-party packages.

in the build.yml workflow on GitHub

nuttx/.github/workflows/build.yml

Line 194 in 201406b

./cibuild.sh -c -A -N -R -S testlist/${{matrix.boards}}.dat

and if desired, you can use it locally
example

./tools/configure.sh -l -S esp32-devkitc:nsh

nuttx/tools/configure.sh

Line 35 in 201406b

-S adds the nxtmpdir folder for third-party packages.

At the moment, simply using CONFIG_ALLOW_DOWNLOADS (must be added to all Expressif configurations!) does not meet the requirements of our CI!

@TangMeng12 Converted back to draft to avoid accidental merging.

The impact must be considered !!!

arch/risc-v/src/esp32c3-legacy/Make.defs

Directly downloading the Git repository is inconvenient for local debugging. This will allow to automatically download external packages from the Internet. If not set, the repo need to be download will need to provide them manually, otherwise an error will occur and the build will be aborted. Add `USE_NXTMPDIR_ESP_REPO_DIRECTLY`, with this we can use `USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make` which can directly use esp-hal-3rdparty under nxtmpdir without CLONE, CHECK_COMMITSHA, reset, checkout and update. Just `cp -rf nxtmpdir/esp-hal-3rdparty chip/$(ESP_HAL_3RDPARTY_REPO)`. Signed-off-by: v-tangmeng <v-tangmeng@xiaomi.com>

tmedicci

Thanks @TangMeng12 . I agree with this solution.

@simbit18 @fdcavalcanti and @eren-terzioglu , can you PTAL?

arch/risc-v/src/common/espressif/Make.defs

Laczen · 2025-12-18T08:14:36Z

@TangMeng12 Could you help me with how to use this feature? I would like to avoid having to download the repository each time after a make distclean is done. Should I manually download the repository to nxtmpdir, or is there some build option that does this? I would like to add some note to the documentation about this feature.

simbit18 · 2025-12-18T10:32:20Z

Hi @Laczen

step used for our CI only for esp32
example

./tools/configure.sh -l -S esp32-devkitc:nsh

-S adds the nxtmpdir folder for third-party packages.

nuttx/tools/configure.sh

Laczen · 2025-12-18T11:50:37Z

Hi @Laczen

step used for our CI only for esp32 example

./tools/configure.sh -l -S esp32-devkitc:nsh

-S adds the nxtmpdir folder for third-party packages.

nuttx/tools/configure.sh

Hi @simbit18, thanks for the reply. I am doing the following:

make distclean
./tools/configure.sh -l -S esp32-devkitc:nsh
make -j 8  --> downloads the repository (almost nothing is added to nxtmpdir/esp-hal-3rdparty: only a tools folder with a script called extract_idf_components.sh).
make distclean
./tools/configure.sh -l -S esp32-devkitc:nsh
make -j 8 --> downloads the repository again

So I think that it is needed to perform an extra step.

fdcavalcanti · 2025-12-18T13:13:39Z

Hi @Laczen
step used for our CI only for esp32 example
./tools/configure.sh -l -S esp32-devkitc:nsh
-S adds the nxtmpdir folder for third-party packages.
nuttx/tools/configure.sh

Hi @simbit18, thanks for the reply. I am doing the following:
make distclean
./tools/configure.sh -l -S esp32-devkitc:nsh
make -j 8  --> downloads the repository (almost nothing is added to nxtmpdir/esp-hal-3rdparty: only a tools folder with a script called extract_idf_components.sh).
make distclean
./tools/configure.sh -l -S esp32-devkitc:nsh
make -j 8 --> downloads the repository again
So I think that it is needed to perform an extra step.

Set the environment variable NXTMPDIR to "../nxtmpdir" and try again.
Something like NXTMPDIR=../ make -j8

simbit18 · 2025-12-18T13:40:15Z

@Laczen test performed now

./tools/configure.sh -l -S esp32-devkitc:nsh

The first time, it creates the nxtmpdir folder and clones esp-hal-3rdparty (https://github.com/espressif/esp-hal-3rdparty.git).

nuttxesp32
       |
       |-nuttx
       |
       |-apps
       |
       |-nxtmpdir
             |
             |-esp-hal-3rdparty

make distclean

The next time, it verifies that the correct esp-hal-3rdparty exists and does not clone it but copies the folder.
Of course, this only applies to this step.

Cloning Espressif HAL for 3rd Party Platforms
Espressif HAL for 3rd Party Platforms: b9472bd56b916cced7447e25c2b2e1390d7e0d90

buntu20042@Unuttx:~/nuttxesp32/nuttx$ make
Create version.h
LN: platform/board to /home/ubuntu20042/nuttxesp32/apps/platform/dummy
Register: dd
Register: nsh
Register: sh
Cloning Espressif HAL for 3rd Party Platforms
Espressif HAL for 3rd Party Platforms: b9472bd56b916cced7447e25c2b2e1390d7e0d90
Espressif HAL for 3rd Party Platforms: initializing submodules...
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 2), reused 2 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (4/4), 87.81 KiB | 908.00 KiB/s, done.
From https://github.com/espressif/esp32-bt-lib
 * branch            6d007b7167146b31256b29fed31e04dc46989885 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 20 (delta 9), reused 15 (delta 4), pack-reused 0 (from 0)
Unpacking objects: 100% (20/20), 85.14 KiB | 871.00 KiB/s, done.
From https://github.com/espressif/esp-coex-lib
 * branch            cb6667841f63457246d56a9170f3409917e9d5a8 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 85, done.
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 48 (delta 29), reused 19 (delta 1), pack-reused 0 (from 0)
Unpacking objects: 100% (48/48), 1.02 MiB | 2.14 MiB/s, done.
From https://github.com/espressif/esp-phy-lib
 * branch            2fb71262baf38dc83591b4191731a0880bfdd3a5 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 115, done.
remote: Counting objects: 100% (115/115), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 67 (delta 54), reused 20 (delta 9), pack-reused 0 (from 0)
Unpacking objects: 100% (67/67), 3.24 MiB | 3.04 MiB/s, done.
From https://github.com/espressif/esp32-wifi-lib
 * branch            7587592417ecdb5163a104aebaedee0a87794608 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 407, done.
remote: Counting objects: 100% (407/407), done.
remote: Compressing objects: 100% (201/201), done.
remote: Total 207 (delta 192), reused 13 (delta 4), pack-reused 0 (from 0)
Receiving objects: 100% (207/207), 334.55 KiB | 1.90 MiB/s, done.
Resolving deltas: 100% (192/192), completed with 188 local objects.
From https://github.com/espressif/mbedtls
 * branch            98ae8db06b8acc1dc1c6b1f78c2719cb776f4147 -> FETCH_HEAD
Applying patches...
CPP:  /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scripts/flat_memory.ld-> /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scripts/flCPP:  /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scripts/esp32_sections.ld-> /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scriptsCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.api.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/CPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/compCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.libc-funcs.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3CPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-reent-funcs.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdpCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/soc/esp32/ld/esp32.peripherals.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/LD: nuttx
Memory region         Used Size  Region Size  %age Used
             ROM:      206824 B    4194272 B      4.93%
     iram0_0_seg:       30116 B       168 KB     17.51%
     irom0_0_seg:      141288 B    3342304 B      4.23%
     dram0_0_seg:       11752 B     180736 B      6.50%
     drom0_0_seg:       77828 B    4194272 B      1.86%
    rtc_iram_seg:           0 B         8 KB      0.00%
    rtc_slow_seg:           0 B         4 KB      0.00%
rtc_reserved_seg:          24 B         24 B    100.00%
      extmem_seg:           0 B         4 MB      0.00%
CP: nuttx.hex

Laczen · 2025-12-18T13:49:36Z

@Laczen test performed now

./tools/configure.sh -l -S esp32-devkitc:nsh

The first time, it creates the nxtmpdir folder and clones esp-hal-3rdparty (https://github.com/espressif/esp-hal-3rdparty.git).

make distclean

The next time, it verifies that the correct esp-hal-3rdparty exists and does not clone it but copies the folder. Of course, this only applies to this step.

Cloning Espressif HAL for 3rd Party Platforms
Espressif HAL for 3rd Party Platforms: b9472bd56b916cced7447e25c2b2e1390d7e0d90

buntu20042@Unuttx:~/nuttxesp32/nuttx$ make
Create version.h
LN: platform/board to /home/ubuntu20042/nuttxesp32/apps/platform/dummy
Register: dd
Register: nsh
Register: sh
Cloning Espressif HAL for 3rd Party Platforms
Espressif HAL for 3rd Party Platforms: b9472bd56b916cced7447e25c2b2e1390d7e0d90
Espressif HAL for 3rd Party Platforms: initializing submodules...
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 2), reused 2 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (4/4), 87.81 KiB | 908.00 KiB/s, done.
From https://github.com/espressif/esp32-bt-lib
 * branch            6d007b7167146b31256b29fed31e04dc46989885 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 20 (delta 9), reused 15 (delta 4), pack-reused 0 (from 0)
Unpacking objects: 100% (20/20), 85.14 KiB | 871.00 KiB/s, done.
From https://github.com/espressif/esp-coex-lib
 * branch            cb6667841f63457246d56a9170f3409917e9d5a8 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 85, done.
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 48 (delta 29), reused 19 (delta 1), pack-reused 0 (from 0)
Unpacking objects: 100% (48/48), 1.02 MiB | 2.14 MiB/s, done.
From https://github.com/espressif/esp-phy-lib
 * branch            2fb71262baf38dc83591b4191731a0880bfdd3a5 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 115, done.
remote: Counting objects: 100% (115/115), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 67 (delta 54), reused 20 (delta 9), pack-reused 0 (from 0)
Unpacking objects: 100% (67/67), 3.24 MiB | 3.04 MiB/s, done.
From https://github.com/espressif/esp32-wifi-lib
 * branch            7587592417ecdb5163a104aebaedee0a87794608 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 407, done.
remote: Counting objects: 100% (407/407), done.
remote: Compressing objects: 100% (201/201), done.
remote: Total 207 (delta 192), reused 13 (delta 4), pack-reused 0 (from 0)
Receiving objects: 100% (207/207), 334.55 KiB | 1.90 MiB/s, done.
Resolving deltas: 100% (192/192), completed with 188 local objects.
From https://github.com/espressif/mbedtls
 * branch            98ae8db06b8acc1dc1c6b1f78c2719cb776f4147 -> FETCH_HEAD
Applying patches...
CPP:  /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scripts/flat_memory.ld-> /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scripts/flCPP:  /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scripts/esp32_sections.ld-> /home/ubuntu20042/nuttxesp32/nuttx/boards/xtensa/esp32/esp32-devkitc/../common/scriptsCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.api.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/CPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/compCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.libc-funcs.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3CPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.newlib-reent-funcs.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdpCPP:  /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/components/soc/esp32/ld/esp32.peripherals.ld-> /home/ubuntu20042/nuttxesp32/nuttx/arch/xtensa/src/chip/esp-hal-3rdparty/LD: nuttx
Memory region         Used Size  Region Size  %age Used
             ROM:      206824 B    4194272 B      4.93%
     iram0_0_seg:       30116 B       168 KB     17.51%
     irom0_0_seg:      141288 B    3342304 B      4.23%
     dram0_0_seg:       11752 B     180736 B      6.50%
     drom0_0_seg:       77828 B    4194272 B      1.86%
    rtc_iram_seg:           0 B         8 KB      0.00%
    rtc_slow_seg:           0 B         4 KB      0.00%
rtc_reserved_seg:          24 B         24 B    100.00%
      extmem_seg:           0 B         4 MB      0.00%
CP: nuttx.hex

@simbit18 Thanks for the reply.

Could you tell me what to expect under /nxtmpdir/esp-hal-3rdparty. I am expecting to see the complete hal with components there so that they are not removed when make distclean is performed again, but I am not seeing that.

Could you verify that your above local test does not redownload after a make distclean is performed.

I am also expecting to see a link to nxtmpdir/esp-hal-3rdparty under ../nuttx/arch/xtensa/src/chip/esp-hal-3rdparty, but
it could be that it still copies at the moment, so this could be a future improvement.

simbit18 · 2025-12-18T14:07:35Z

@Laczen as mentioned earlier, option -S
only clones esp-hal-3rdparty into the nxtmpdir folder.

make distclean removes esp-hal-3rdparty (only the copied one, of course :) -> chip/$(ESP_HAL_3RDPARTY_REPO) ) with all subsequent steps after cloning.

ifneq ($(USE_NXTMPDIR_ESP_REPO_DIRECTLY),y)
	$(Q) echo ‘Espressif HAL for 3rd Party Platforms: ${ESP_HAL_3RDPARTY_VERSION}’
    $(Q) git -C chip/$(ESP_HAL_3RDPARTY_REPO) checkout --quiet $(ESP_HAL_3RDPARTY_VERSION)
	$(Q) git -C chip/$(ESP_HAL_3RDPARTY_REPO) submodule --quiet update --init $(GIT_DEPTH_PARAMETER) components/mbedtls/mbedtls
endif
ifeq ($(CONFIG_ESP_WIRELESS),y)
ifneq ($(USE_NXTMPDIR_ESP_REPO_DIRECTLY),y)
    $(Q) echo ‘Espressif HAL for 3rd Party Platforms: initialising submodules...’
	$(Q) git -C chip/$(ESP_HAL_3RDPARTY_REPO) submodule --quiet update --init $(GIT_DEPTH_PARAMETER) components/esp_phy/lib components/esp_wifi/lib components/bt/controller/lib_esp32c3_family components/esp_coex/lib
	$(Q) git -C chip/$(ESP_HAL_3RDPARTY_REPO)/components/mbedtls/mbedtls reset --quiet --hard
endif
    $(Q) echo ‘Applying patches...’
	$(Q) cd chip/$(ESP_HAL_3RDPARTY_REPO)/components/mbedtls/mbedtls && git apply ../../../nuttx/patches/components/mbedtls/mbedtls/*.patch
endif

Espressif HAL for 3rd Party Platforms: initializing submodules...
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 2), reused 2 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (4/4), 87.81 KiB | 908.00 KiB/s, done.
From https://github.com/espressif/esp32-bt-lib
 * branch            6d007b7167146b31256b29fed31e04dc46989885 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 20 (delta 9), reused 15 (delta 4), pack-reused 0 (from 0)
Unpacking objects: 100% (20/20), 85.14 KiB | 871.00 KiB/s, done.
From https://github.com/espressif/esp-coex-lib
 * branch            cb6667841f63457246d56a9170f3409917e9d5a8 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 85, done.
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 48 (delta 29), reused 19 (delta 1), pack-reused 0 (from 0)
Unpacking objects: 100% (48/48), 1.02 MiB | 2.14 MiB/s, done.
From https://github.com/espressif/esp-phy-lib
 * branch            2fb71262baf38dc83591b4191731a0880bfdd3a5 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 115, done.
remote: Counting objects: 100% (115/115), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 67 (delta 54), reused 20 (delta 9), pack-reused 0 (from 0)
Unpacking objects: 100% (67/67), 3.24 MiB | 3.04 MiB/s, done.
From https://github.com/espressif/esp32-wifi-lib
 * branch            7587592417ecdb5163a104aebaedee0a87794608 -> FETCH_HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Enumerating objects: 407, done.
remote: Counting objects: 100% (407/407), done.
remote: Compressing objects: 100% (201/201), done.
remote: Total 207 (delta 192), reused 13 (delta 4), pack-reused 0 (from 0)
Receiving objects: 100% (207/207), 334.55 KiB | 1.90 MiB/s, done.
Resolving deltas: 100% (192/192), completed with 188 local objects.
From https://github.com/espressif/mbedtls
 * branch            98ae8db06b8acc1dc1c6b1f78c2719cb776f4147 -> FETCH_HEAD
Applying patches...

so only the clone phase is skipped

Laczen · 2025-12-18T14:24:57Z

@simbit18 probably my expectations are wrong.

What I see under /nxtmpdir/esp-hal-3rdparty is the main branch of espressif/esp-hal-3rdparty. The main branch only contains a tools directory (with one script file) and a LICENSE and README.md.

I would expect to see the release/master.a (or similar) branch instead. At the moment it is not clear for me why the clone of the main branch is performed.

simbit18 · 2025-12-18T14:49:03Z

@Laczen This is the system adopted by Espressif.

However, the folder is a total of 300 MB.
If you try to imagine how many times it would need to be cloned in our CI (all esp32, S2, S3, C3, C6 ecc boards:config), you will understand why the -S option was added.

Laczen · 2025-12-18T18:23:10Z

@simbit18 please don't take my question as criticism on the chosen solution. Not only the espressif CI could benefit from the chosen solution. If a small change/better user understanding would allow avoiding downloads after make distclean the feature would benefit many users. As far as I understand the PR the intention is to avoid downloads by using USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make. For me this doesn`t work, so I must be missing some step.

Laczen · 2025-12-18T19:20:22Z

@simbit18 I have been able to get it working by only checking out the release/master.a branch to nxtmpdir, initializing the submodules in the nxtmpdir, applying the patches in nxtmpdir and change the Make.def to only applying the patches if USE_NXTMPDIR_ESP_REPO_DIRECTLY=n. Then the build succeeds with USE_NXTMPDIR_ESP_REPO_DIRECTLY=y make.

IMHO it is also possible to replace the copying of files with a link (at least on linux).

Laczen · 2025-12-19T13:05:34Z

@simbit18, with one very small change in Config.mk line 638: cp -fr $3 $2; \ into ln -sf $3 $2; \ the build after a distclean can be performed without any additional downloads.

TangMeng12 · 2025-12-23T06:10:35Z

./tools/configure.sh -l -S esp32-devkitc:nsh

Step 1: Download and initialize your local esp-hal-3rdparty repository, including the following:
① Download esp-hal-3rdparty to the nxtmpdir directory
② Initialize the esp-hal-3rdparty submodule in nxtmpdir
Step 2: Set USE_NXTMPDIR_ESP_REPO_DIRECTLY=y
Step 3: ./tools/configure.sh -S esp32-devkitc:nsh
This will directly use the esp-hal-3rdparty code in the nxtmpdir directory without performing any download or submodule initialization (it will still download code).

The reasons are as follows:

The -S option specifies using the nxtmpdir directory and reusing the repository in this directory.
USE_NXTMPDIR_ESP_REPO_DIRECTLY=y will skip checking the commit ID information of esp-hal-3rdparty , and skip the checkout and submodule initialization actions.
If the local repository is missing content, there may be compilation issues.

github-actions bot added Arch: risc-v Issues related to the RISC-V (32-bit or 64-bit) architecture Arch: xtensa Issues related to the Xtensa architecture Size: M The size of the change in this PR is medium labels Oct 23, 2025

TangMeng12 force-pushed the master branch 2 times, most recently from 05d05e7 to eac577d Compare October 23, 2025 12:04

xiaoxiang781216 previously approved these changes Oct 23, 2025

View reviewed changes

TangMeng12 marked this pull request as draft October 23, 2025 13:06

TangMeng12 dismissed xiaoxiang781216’s stale review via 9e457f2 October 23, 2025 13:06

TangMeng12 force-pushed the master branch from eac577d to 9e457f2 Compare October 23, 2025 13:06

github-actions bot added the Size: L The size of the change in this PR is large label Oct 23, 2025

TangMeng12 force-pushed the master branch 3 times, most recently from ed60e98 to 067398b Compare October 23, 2025 14:09

simbit18 requested review from fdcavalcanti and tmedicci October 23, 2025 14:44

TangMeng12 force-pushed the master branch from 067398b to f6acbbf Compare October 27, 2025 07:15

github-actions bot added the Size: S The size of the change in this PR is small label Oct 27, 2025

TangMeng12 force-pushed the master branch from f6acbbf to ef9e173 Compare October 27, 2025 08:08

TangMeng12 force-pushed the master branch 2 times, most recently from 73db8b4 to 826faa4 Compare October 27, 2025 10:08

TangMeng12 marked this pull request as ready for review October 27, 2025 12:18

TangMeng12 requested review from Donny9 and raiden00pl as code owners October 27, 2025 12:18

simbit18 marked this pull request as draft October 27, 2025 13:58

simbit18 requested a review from lupyuen October 27, 2025 14:00

TangMeng12 force-pushed the master branch from 5df2468 to 38af81a Compare November 11, 2025 02:50

github-actions bot added Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces. and removed Area: Tooling Area: Build system labels Nov 11, 2025

TangMeng12 force-pushed the master branch from 38af81a to fd34d7b Compare November 11, 2025 02:51

TangMeng12 marked this pull request as ready for review November 11, 2025 03:26

tmedicci reviewed Nov 14, 2025

View reviewed changes

arch/risc-v/src/esp32c3-legacy/Make.defs Outdated Show resolved Hide resolved

TangMeng12 force-pushed the master branch from fd34d7b to 57f0609 Compare November 18, 2025 03:38

tmedicci approved these changes Nov 18, 2025

View reviewed changes

fdcavalcanti reviewed Nov 18, 2025

View reviewed changes

arch/risc-v/src/common/espressif/Make.defs Show resolved Hide resolved

eren-terzioglu approved these changes Nov 18, 2025

View reviewed changes

fdcavalcanti approved these changes Nov 18, 2025

View reviewed changes

tmedicci requested review from acassis and xiaoxiang781216 November 18, 2025 13:21

xiaoxiang781216 approved these changes Nov 18, 2025

View reviewed changes

xiaoxiang781216 merged commit a221df7 into apache:master Nov 18, 2025
40 checks passed

arch/xtensa/esp32[-s2|-s3]: Modify the method of downloading the repository #17236

arch/xtensa/esp32[-s2|-s3]: Modify the method of downloading the repository #17236

Uh oh!

Conversation

TangMeng12 commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Impact

Testing

Uh oh!

simbit18 commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hartmannathan commented Oct 26, 2025

Uh oh!

Laczen commented Oct 27, 2025

Uh oh!

TangMeng12 commented Oct 27, 2025

Uh oh!

simbit18 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tmedicci left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Laczen commented Dec 18, 2025

Uh oh!

simbit18 commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Laczen commented Dec 18, 2025

Uh oh!

fdcavalcanti commented Dec 18, 2025

Uh oh!

simbit18 commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Laczen commented Dec 18, 2025

Uh oh!

simbit18 commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Laczen commented Dec 18, 2025

Uh oh!

simbit18 commented Dec 18, 2025

Uh oh!

Laczen commented Dec 18, 2025

Uh oh!

Laczen commented Dec 18, 2025

Uh oh!

Laczen commented Dec 19, 2025

Uh oh!

TangMeng12 commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

TangMeng12 commented Oct 23, 2025 •

edited

Loading

simbit18 commented Oct 23, 2025 •

edited

Loading

simbit18 commented Oct 27, 2025 •

edited

Loading

simbit18 commented Dec 18, 2025 •

edited

Loading

simbit18 commented Dec 18, 2025 •

edited

Loading

simbit18 commented Dec 18, 2025 •

edited

Loading

TangMeng12 commented Dec 23, 2025 •

edited

Loading