tools/espressif: add esptool version check to Espressif build system #14267

fdcavalcanti · 2024-10-14T10:58:26Z

Summary

Add a Python script to check if esptool is installed and if its version is up-to-date (see minimum version at ESPTOOL_MIN_VERSION on .mk files).

This fixes build issues as reported in #13824.

Impact

New Python script added. Stops build if esptool version is below 4.8.
Script runs for ESP32, S2, S3, C3, C6.

Testing

Tested locally and on internal CI with Python3.8+.

nuttxpr · 2024-10-14T11:00:51Z

[Experimental Bot, please feedback here]

Fill In The Commit Message: This PR contains a Commit with an Empty Commit Message. Please fill in the Commit Message with the PR Summary.

The PR summary and impact sections appear to meet NuttX requirements. However, the testing section is insufficient.

Here's what needs improvement:

Testing details: The provided information is too vague.
- Be specific about build hosts: Specify the exact OS versions, CPU architectures, and compiler versions used for testing. For example, "Ubuntu 20.04, x86_64, GCC 9.4" is much better than just "Linux".
- Be specific about target(s): Clearly list each tested architecture, board, and configuration. Don't just say "ESP32" - specify "ESP32-DevKitC" or similar.
- Show actual logs: The "Testing logs before/after" sections are empty. Include relevant snippets demonstrating the issue before the change and the successful outcome after.

In short, provide concrete evidence that your change works as intended.

jerpelea · 2024-10-14T12:11:40Z

12.7 release backport #14268

jerpelea · 2024-10-14T13:50:37Z

@fdcavalcanti this PR should target master. The back-port targets 12.7

Please rebase

fdcavalcanti · 2024-10-14T14:46:12Z

@jerpelea rebased. Please verify.

cederom · 2024-10-14T17:05:21Z

Thank you @fdcavalcanti !! Works fine :-)

Two minor python script tweaks to consider:

Put interpreter string on the first line.
Add expected version to error message, see review comment, example provided.

Testing on FreeBSD with esptool 4.7.0:

% ./tools/configure.sh -B esp32-devkitc:nsh
% gmake -j8
Create version.h
LN: platform/board to /XXX/nuttx-apps.git/platform/dummy
Cloning Espressif HAL for 3rd Party Platforms
Clone: chip/esp-hal-3rdparty Register: nsh
Register: sh
Espressif HAL for 3rd Party Platforms: b4c723a119344b4b71d69819019d55637fb570fd
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
Unsupported esptool version: 4.7.0
Upgrade using: 'pip install --upgrade esptool' and run 'make' again
gmake: *** [tools/Unix.mk:558: nuttx] Error 1

With my suggestion on expected version added:

% gmake -j8
Create version.h
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
Unsupported esptool version 4.7.0 expected >= 4.8
Upgrade using: 'pip install --upgrade esptool' and run 'make' again
gmake: *** [tools/Unix.mk:558: nuttx] Error 1

With esptool 4.8.1 from venv works fine:

(venv3.9embedded) gmake -j8
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.8.1
Creating esp32 image...
Image has only RAM segments visible. ROM segments are hidden and SHA256 digest is not appended.
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin

(venv3.9embedded) gmake flash
LD: nuttx
CP: nuttx.hex
MKIMAGE: ESP32 binary
esptool.py -c esp32 elf2image --ram-only-header -fs 4MB -fm dio -ff 40m -o nuttx.bin nuttx
esptool.py v4.8.1
Creating esp32 image...
Image has only RAM segments visible. ROM segments are hidden and SHA256 digest is not appended.
Merged 1 ELF section
Successfully created esp32 image.
Generated: nuttx.bin
esptool.py -c esp32 -p /dev/cuaU0 -b 115200  write_flash -fs detect -fm dio -ff 40m 0x1000 nuttx.bin
esptool.py v4.8.1
Serial port /dev/cuaU0
Connecting..........
Chip is ESP32-D0WD-V3 (revision v3.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: XXX
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00001000 to 0x00032fff...
Compressed 203204 bytes to 74440...
Wrote 203204 bytes (74440 compressed) at 0x00001000 in 7.0 seconds (effective 232.4 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

(venv3.9embedded) cu -l /dev/cuaU0 -s 115200
Connected
ts Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3ffb1f50,len:1004
ho 0 tail 12 room 4
load:0x40080000,len:20708
entry 0x40082610
*** Booting NuttX ***
dram: lma 0x00001020 vma 0x3ffb1f50 len 0x3ec    (1004)
iram: lma 0x00001414 vma 0x40080000 len 0x50e4   (20708)
padd: lma 0x00006508 vma 0x00000000 len 0x9af0   (39664)
imap: lma 0x00010000 vma 0x400e0000 len 0x11598  (71064)
padd: lma 0x000215a0 vma 0x00000000 len 0xea78   (60024)
dmap: lma 0x00030020 vma 0x3f400020 len 0x29a4   (10660)
total segments stored 6

NuttShell (NSH) NuttX-12.7.0-RC0
nsh> uname -a
NuttX 12.7.0-RC0 cfba4e334a-dirty Oct 14 2024 19:00:57 xtensa esp32-devkitc

cederom

Thank you @fdcavalcanti :-) Please update python script, examples provided :-)

tools/espressif/check_esptool.py

jerpelea · 2024-10-15T07:37:35Z

@fdcavalcanti please update so that we can merge it and unblock the release (I will pick it to 12.7)

fdcavalcanti · 2024-10-15T08:25:43Z

@jerpelea updated with expected version. Thanks.

jerpelea · 2024-10-15T10:55:19Z

@fdcavalcanti something gone really wrong

jerpelea · 2024-10-15T11:07:13Z

@lupyuen
../nuttx/tools/checkpatch.sh: line 86: black: command not found
../nuttx/tools/checkpatch.sh: line 87: flake8: command not found
../nuttx/tools/checkpatch.sh: line 88: isort: command not found

fdcavalcanti · 2024-10-15T11:10:02Z

@lupyuen ../nuttx/tools/checkpatch.sh: line 86: black: command not found ../nuttx/tools/checkpatch.sh: line 87: flake8: command not found ../nuttx/tools/checkpatch.sh: line 88: isort: command not found

Seems the Python linters are not installed in the CI test virtual environment. The rest looks good.

lupyuen · 2024-10-15T11:13:13Z

Seems the Python linters are not installed in the CI test virtual environment. The rest looks good.

@fdcavalcanti So do we need to install them in the Dockerfile right now? https://github.com/apache/nuttx/blob/master/tools/ci/docker/linux/Dockerfile

Update: Oops this is the Python Virtual Environment, not the Docker Container

lupyuen · 2024-10-15T11:18:46Z

@fdcavalcanti Do we need this to fix black etc? #14253

    black --check $@ || fail=1

Update: Hmmm nope, I already see it here: https://github.com/fdcavalcanti/nuttx/blob/feature/esptool_script_check_min_version/tools/checkpatch.sh#L86

fdcavalcanti · 2024-10-15T11:26:58Z

@fdcavalcanti Do we need this to fix black etc? #14253
    black --check $@ || fail=1

It just needs to be installed in a venv, similar to this

lupyuen · 2024-10-15T11:29:58Z

@fdcavalcanti The venv is already the latest version I think? I see this in the log below. Where is it picking up the black to be installed? (Very sorry I'm awful in Python)

Run echo "::add-matcher::nuttx/.github/nxstyle.json"
  echo "::add-matcher::nuttx/.github/nxstyle.json"
  python -m venv .venv
  source .venv/bin/activate
  pip install cmake-format
  cd nuttx
  commits="8f3060ed5[2](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:2)2bcd143bb987a44ebd6ffd4da9a85c..HEAD"
  git log --oneline $commits
  echo "../nuttx/tools/checkpatch.sh -u -m -g $commits"
  ../nuttx/tools/checkpatch.sh -u -m -g $commits
  shell: /usr/bin/bash -e {0}
Collecting cmake-format
  Downloading cmake_format-0.6.1[3](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:3)-py3-none-any.whl.metadata (23 kB)
Collecting cmakelang>=0.6.13 (from cmake-format)
  Downloading cmakelang-0.6.13-py3-none-any.whl.metadata (23 kB)
Collecting six>=1.13.0 (from cmakelang>=0.6.13->cmake-format)
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Downloading cmake_format-0.6.13-py3-none-any.whl (19 kB)
Downloading cmakelang-0.6.13-py3-none-any.whl (159 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 159.8/159.8 kB 11.7 MB/s eta 0:00:00
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, cmakelang, cmake-format
Successfully installed cmake-format-0.6.13 cmakelang-0.6.13 six-1.16.0
e[4](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:4)7908f1b1 Merge a9593597841d2ee0942cb[5](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:5)19f7b9242e2335657c into 8f30[6](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:6)0ed522bcd143bb98[7](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:7)a44ebd6ffd4da9a85c
a959359784 tools/espressif: add esptool version check to Espressif build system
../nuttx/tools/checkpatch.sh -u -m -g [8](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:8)f3060ed522bcd143bb987a44ebd6ffd4da[9](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:9)a85c..HEAD
../nuttx/tools/checkpatch.sh: line 86: black: command not found
../nuttx/tools/checkpatch.sh: line 87: flake8: command not found
../nuttx/tools/checkpatch.sh: line 88: isort: command not found

https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267

fdcavalcanti · 2024-10-15T11:34:09Z

@fdcavalcanti The venv is already the latest version I think? I see this in the log below. Where is it picking up the black to be installed? (Very sorry I'm awful in Python)

Run echo "::add-matcher::nuttx/.github/nxstyle.json"
  echo "::add-matcher::nuttx/.github/nxstyle.json"
  python -m venv .venv
  source .venv/bin/activate
  pip install cmake-format
  cd nuttx
  commits="8f3060ed5[2](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:2)2bcd143bb987a44ebd6ffd4da9a85c..HEAD"
  git log --oneline $commits
  echo "../nuttx/tools/checkpatch.sh -u -m -g $commits"
  ../nuttx/tools/checkpatch.sh -u -m -g $commits
  shell: /usr/bin/bash -e {0}
Collecting cmake-format
  Downloading cmake_format-0.6.1[3](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:3)-py3-none-any.whl.metadata (23 kB)
Collecting cmakelang>=0.6.13 (from cmake-format)
  Downloading cmakelang-0.6.13-py3-none-any.whl.metadata (23 kB)
Collecting six>=1.13.0 (from cmakelang>=0.6.13->cmake-format)
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Downloading cmake_format-0.6.13-py3-none-any.whl (19 kB)
Downloading cmakelang-0.6.13-py3-none-any.whl (159 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 159.8/159.8 kB 11.7 MB/s eta 0:00:00
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, cmakelang, cmake-format
Successfully installed cmake-format-0.6.13 cmakelang-0.6.13 six-1.16.0
e[4](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:4)7908f1b1 Merge a9593597841d2ee0942cb[5](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:5)19f7b9242e2335657c into 8f30[6](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:6)0ed522bcd143bb98[7](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:7)a44ebd6ffd4da9a85c
a959359784 tools/espressif: add esptool version check to Espressif build system
../nuttx/tools/checkpatch.sh -u -m -g [8](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:8)f3060ed522bcd143bb987a44ebd6ffd4da[9](https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267#step:3:9)a85c..HEAD
../nuttx/tools/checkpatch.sh: line 86: black: command not found
../nuttx/tools/checkpatch.sh: line 87: flake8: command not found
../nuttx/tools/checkpatch.sh: line 88: isort: command not found

https://github.com/apache/nuttx/actions/runs/11344800359/job/31550311398?pr=14267

Its only pip installing cmake (right after enabling the venv), need to also install black, isort, flake8.
We can move this discussion elsewhere, if you prefer.

lupyuen · 2024-10-15T11:38:16Z

@fdcavalcanti Not sure if I can help much? I might make things worse (since I know nothing about Python)

@acassis @cederom Do you know who can help with this Python Environment issue? (Simbit is on holiday I think)

jerpelea · 2024-10-15T13:01:31Z

@fdcavalcanti please rebase on master

fdcavalcanti · 2024-10-15T15:04:22Z

@fdcavalcanti please rebase on master

We're good.

cederom · 2024-10-15T22:46:19Z

Sorry I was out today on a funeral in a different city.

Btw we have esptool 4.8.1 in FreeBSD system packages now: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282000.

Thank you for the fix!!

github-actions bot added Area: Tooling Size: M The size of the change in this PR is medium labels Oct 14, 2024

fdcavalcanti force-pushed the feature/esptool_script_check_min_version branch 2 times, most recently from 89223ec to 9127152 Compare October 14, 2024 11:59

jerpelea requested a review from cederom October 14, 2024 12:53

jerpelea approved these changes Oct 14, 2024

View reviewed changes

acassis approved these changes Oct 14, 2024

View reviewed changes

jerpelea changed the base branch from releases/12.7 to master October 14, 2024 13:51

jerpelea changed the base branch from master to releases/12.7 October 14, 2024 13:51

fdcavalcanti changed the base branch from releases/12.7 to master October 14, 2024 14:37

fdcavalcanti force-pushed the feature/esptool_script_check_min_version branch from 9127152 to cfba4e3 Compare October 14, 2024 14:45

cederom requested changes Oct 14, 2024

View reviewed changes

tools/espressif/check_esptool.py Outdated Show resolved Hide resolved

tools/espressif/check_esptool.py Show resolved Hide resolved

cederom mentioned this pull request Oct 14, 2024

{Bp-14267} tools/espressif: add esptool version check to Espressif build system #14268

Merged

xiaoxiang781216 approved these changes Oct 14, 2024

View reviewed changes

fdcavalcanti force-pushed the feature/esptool_script_check_min_version branch from cfba4e3 to 0208695 Compare October 15, 2024 08:24

fdcavalcanti force-pushed the feature/esptool_script_check_min_version branch from 0208695 to 92f8f0c Compare October 15, 2024 11:04

fdcavalcanti force-pushed the feature/esptool_script_check_min_version branch from 92f8f0c to a959359 Compare October 15, 2024 11:08

tools/espressif: add esptool version check to Espressif build system

e01212a

fdcavalcanti force-pushed the feature/esptool_script_check_min_version branch from a959359 to e01212a Compare October 15, 2024 14:54

jerpelea merged commit 5592145 into apache:master Oct 15, 2024

This was referenced Oct 15, 2024

Update esptool version on docs and build system #14120

Merged

Revert "Update esptool version on docs and build system" #14338

Merged

tools/espressif: add esptool version check to Espressif build system #14267

tools/espressif: add esptool version check to Espressif build system #14267

Uh oh!

Conversation

fdcavalcanti commented Oct 14, 2024

Summary

Impact

Testing

Uh oh!

nuttxpr commented Oct 14, 2024

Uh oh!

jerpelea commented Oct 14, 2024

Uh oh!

jerpelea commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fdcavalcanti commented Oct 14, 2024

Uh oh!

cederom commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cederom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jerpelea commented Oct 15, 2024

Uh oh!

fdcavalcanti commented Oct 15, 2024

Uh oh!

jerpelea commented Oct 15, 2024

Uh oh!

jerpelea commented Oct 15, 2024

Uh oh!

fdcavalcanti commented Oct 15, 2024

Uh oh!

lupyuen commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lupyuen commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fdcavalcanti commented Oct 15, 2024

Uh oh!

lupyuen commented Oct 15, 2024

Uh oh!

fdcavalcanti commented Oct 15, 2024

Uh oh!

lupyuen commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jerpelea commented Oct 15, 2024

Uh oh!

fdcavalcanti commented Oct 15, 2024

Uh oh!

cederom commented Oct 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jerpelea commented Oct 14, 2024 •

edited

Loading

cederom commented Oct 14, 2024 •

edited

Loading

lupyuen commented Oct 15, 2024 •

edited

Loading

lupyuen commented Oct 15, 2024 •

edited

Loading

lupyuen commented Oct 15, 2024 •

edited

Loading