Arduino ZModem for LittleFS and Raspberry Pico

[git2fa]

Hello and thank you for your nice project.

I am searching for a nice technique to send and receive files to an Raspberry Pico and doing it over the already implemented USB UART would be an good solution.
Currently i am using this RP2040 Dual-Core ARM Cortex-M0+ Processor with 264 KB RAM and 16 MB Flash, so a 15 MB space on the flash for data is available.

Do you already have experience with the use of LittleFS or is there an ready implementation for the use with ZModem for it?

[git2fa]

Now I see that there is already an implementation for getting files using gdb:

https://arduino-pico.readthedocs.io/en/latest/fs.html

But this needs different tools and does not support different file commands like delete file.
A direct communication via a client that can use ZModem would be much more convenient.

[ecm-bitflipper]

I don't personally have any experience with LittleFS. The sketch as it stands today uses SDFS. Sounds like your board is large enough to handle the existing sketch and memory requirements, why not just try it with SDFS?

If you really wanted to adapt the code to use LittleFS it wouldn't be an enormous amount of work. Most of the code is Z-Modem spaghetti. You would just have to locate everywhere that the SDFS objects are used (unfortunately I couldn't keep them all centralized in one source file without rewriting the whole thing and likely breaking it) and adapt those.

[git2fa]

Thank you for your response and statement.
It was just a spontaneous impulse to ask if there is any further experience available. :-)
I will see if the software can be adapted to LittleFS and will report the results here.

Of course it is senseful to try to compile your code first.
Here are the main errors:

/tmp/arduino_build_152654/sketch/zmodem_sz.cpp: In function 'int wctx(long int)':
zmodem_fixes.h:31:59: error: 'SERIAL_TX_BUFFER_SIZE' was not declared in this scope
   31 | #define sendline(_c) ({ if (ZSERIAL.availableForWrite() > SERIAL_TX_BUFFER_SIZE / 2) ZSERIAL.flush(); ZSERIAL.write(char(_c)); })
      |                                                           ^~~~~~~~~~~~~~~~~~~~~

../Arduino/Pico-ZModem/Pico-ZModem.ino: In function 'size_t DSERIALprint(const arduino::__FlashStringHelper*)':
Pico-ZModem:162:39: error: 'SERIAL_TX_BUFFER_SIZE' was not declared in this scope
  162 |     if (DSERIAL.availableForWrite() > SERIAL_TX_BUFFER_SIZE / 2) DSERIAL.flush();
      |                                       ^~~~~~~~~~~~~~~~~~~~~

../Arduino/Pico-ZModem/Pico-ZModem.ino: In function 'int count_files(int*, long int*)':
Pico-ZModem:248:8: error: 'class FsFile' has no member named 'openCwd'; did you mean 'open'?
  248 |   dir->openCwd();
      |        ^~~~~~~

../Arduino/Pico-ZModem/Pico-ZModem.ino: In function 'void loop()':
Pico-ZModem:292:13: error: ordered comparison of pointer with integer zero ('char*' and 'int')
  292 |   if (param > 0) {
      |       ~~~~~~^~~

Pico-ZModem:311:10: error: 'class FsFile' has no member named 'openCwd'; did you mean 'open'?
  311 |     dir->openCwd();
      |          ^~~~~~~

zmodem_fixes.h:31:59: error: 'SERIAL_TX_BUFFER_SIZE' was not declared in this scope
   31 | #define sendline(_c) ({ if (ZSERIAL.availableForWrite() > SERIAL_TX_BUFFER_SIZE / 2) ZSERIAL.flush(); ZSERIAL.write(char(_c)); })
      |                                                           ^~~~~~~~~~~~~~~~~~~~~

Where should the buffer size for ZMODEM and the Serial interface be declared?

[git2fa]

Adding a defintion for SERIAL_TX_BUFFER_SIZE in zmodem_config.h reduces the errors to the file library.

// Dylan (monte_carlo_ecm, bitflipper, etc.) - Adjust the baud rate to suit your board and needs
#define ZMODEM_SPEED 57600
#define SERIAL_TX_BUFFER_SIZE 100

Errors now:

../Arduino/Pico-ZModem/Pico-ZModem.ino: In function 'int count_files(int*, long int*)':
Pico-ZModem:248:8: error: 'class FsFile' has no member named 'openCwd'; did you mean 'open'?
  248 |   dir->openCwd();
      |        ^~~~~~~
      |        open
../Arduino/Pico-ZModem/Pico-ZModem.ino: In function 'void loop()':
Pico-ZModem:292:13: error: ordered comparison of pointer with integer zero ('char*' and 'int')
  292 |   if (param > 0) {
      |       ~~~~~~^~~
Pico-ZModem:311:10: error: 'class FsFile' has no member named 'openCwd'; did you mean 'open'?
  311 |     dir->openCwd();
      |          ^~~~~~~
      |          open
Pico-ZModem:334:10: error: 'class FsFile' has no member named 'openCwd'; did you mean 'open'?
  334 |     dir->openCwd();
      |          ^~~~~~~
      |          open
Pico-ZModem:394:15: error: 'class FsFile' has no member named 'openCwd'; did you mean 'open'?
  394 |         dirsz.openCwd();
      |               ^~~~~~~

Now it makes sense to have a closer look to the use of the file system and replace it to LittleFS.

[git2fa]

Up to now it is not clear what the difference between DSERIAL and ZSERIAL is?
It seems ZSERIAL is only for Debug purposes, or must it always be active?
Cleaning the serial interface stuff seems to be the first big task before entering the file system stuff.
For other MCU's the Arduino implementation of the UART is a simplification of the HAL interface for advanced 32-Bit MCU's.

Generally it is not easy to come into the line of programming of such foreign code ...

[ecm-bitflipper]

I believe SERIAL_TX_BUFFER_SIZE is 64 for most boards. Try that, or at least a power of 2. 100 might give you trouble.

if (param > 0) {

The compiler is getting fussier over the years. Change it to this;

if ((int)param > 0) {

For the openCwd() thing, you are using a different version of the SD library than me. I believe I used this one.

https://github.com/greiman/SdFat

openCwd is a valid class member in that version, I checked the code.

The ZSERIAL DSERIAL thing, you're having me think back many years. I know I was using an Arduino emulator on a PC to build this library. I had two serial ports configured in there, one for the actual transfer and the other for debug info. I'd say just leave it alone, if I remember correctly I removed all the debug outputs prior to releasing the library. Other folks have used it successfully anyway.

I hope that helps you make a step forward.
Dylan.

[git2fa]

Thank you Dylan, now some things make more sense.

Please don't be disappointed, but this sub-project is taking much more time than expected.
My programming skills are not so good and normally I work more with KISS (keep it simple & stupid).
I guess this sub-project needs for me at least 2 more days to be rewritten and debugged.

At this time it is more important to get the main source in a project working and there is already a simple command line structure implemented. So it is much less effort to write a simple dump of configuration files and logged CSV data.
Additional deleting a file in a list is very quick and easy.

Maybe I will later come back and try to implement the ZMODEM stuff additional. Then I will post again.
Here is the last state of the files including the compiler output (Please download and save as Pico-ZModem.zip).
Maybe it is useful for someone else.

The SdFat library has already been used for data loggers with an STM32 MCU.
(Now you get the above Pico for the same price below $2.)
But the main advantage of an SD-card is that it can be simple extracted and connected to an PC to read and modify data.

Have you read the comment at https://github.com/greiman/SdFat?

Warning: This version has major internal changes.
SdFat version 2.3.0 has major changes to implement RP2040/RP2350 SDIO.

[ecm-bitflipper]

Okay, no worries, good luck with what you are working on.

[git2fa]

Nevertheless, it's not so easy to put it aside completely once you have started it. :-)
It's not needed to make the whole example work.
True to the motto of KISS, I will make a short test if it's possible to compile and start an upload and download via ZModem of a test file.

[git2fa]

Tried it once more now, but it is really not easy, because it seems that <SdFat.h> is included, even when it is commented out everywhere.

Additional there is no standard method in LittleFS to read a specified number of bytes without #include <VFS.h>
https://arduino-pico.readthedocs.io/en/latest/fs.html
But this is not the problem, because linking fails even it is commented out.

There are no compile errors, but it cannot be linked and there are this error messages:

../Arduino/libraries/SdFat/src/FsLib/FsFile.h:365:(.text._Z10procheaderPc+0xa4): undefined reference to `fout'
/srv/bin/arduino15/packages/rp2040/tools/pqt-gcc/4.0.1-8ec9d6f/bin/../lib/gcc/arm-none-eabi/14.2.0/../../../../arm-none-eabi/bin/ld: /tmp/arduino_build_96165/sketch/zmodem_rz.cpp.o:/win/AVR-GCC/Arduino/libraries/SdFat/src/ExFatLib/ExFatFile.h:251: more undefined references to `fout' follow
collect2: error: ld returned 1 exit status

Here is the last state of the files including the compiler output (Please download and save as Pico-ZModem.zip).

[ecm-bitflipper]

The only reason I can see for that is that the newer compiler is getting confused by the strange way I declared fout. Try this.

In zmodem.ino, you'll find a line

extern SdFile fout;

Get rid of the extern on that line. Then later in the same file you'll find

SdFile fout;

Get rid of that line altogether. Try compiling.

Dylan.

[git2fa]

The 3 lines are commented out in Pico-ZModem.ino the whole time.

// #include <SdFat.h>
// #include <SdFatUtil.h>
// SdFs sd;

Yesterday I already did have the idea that the reason is, that the object for fout is only declared in Pico-ZModem.ino and not declared in zmodem_sz.cpp and zmodem_rz.cpp and so the linker cannot connect it.
(Besides #include <VFS.h> has never been used, because no function calls have happened.)

There are two possible solutions:
a) The file object must be handled to the library files (but I never done this before).
b) All file activity must be done within one file, so that the file object is known.

At least b) is the better and clearer solution, because ZModem functionality should be condensed in one library, that can be used and configured in one additional header file. But this needs time again ...

[ecm-bitflipper]

I think you are making the problem bigger than it is. Did you try the simple 2 line change that I recommended?

[git2fa]

It is the task to use LittleFS within the flash chip of the MCU and not SDFat for an external SD card.

[ecm-bitflipper]

Okay, for that you are embarking on new territory. I recommend you cut a new branch and go from there. It will take you some time but it is likely do-able. Good luck with it!