drivers/mtd/ftl/bch: Add direct writing method for FTL

[jingfei195887]

Summary

1. The registration method for MTD devices in nuttx/boards have been adjusted , replacing the previous approach using ftl_initialize() andbchdev_register()with register_mtddriver().When registering MTD devices via register_mtddriver(), FTL and BCH wrappers will be added during the open() process:

   • When accessing the MTD device node via the open() interface, the device will be automatically converted to a character device. Both FTL and BCH wrappers will be implicitly added, provided that BCH support is enabled in the configuration.

   • When accessing the MTD device node via open_blockdriver(), the device will be treated as a block device, with only the FTL wrapper automatically applied.

   Due to the automatic wrapping of MTD devices during the open() process, the legacy registration methods
   ftl_initialize() and bchdev_register() are no longer required for MTD device registration for user code and should
   be used only internally within fs and driver code.

   The functions register_partition_with_mtd() and register_mtdpartition() are actually wrappers built on top of register_mtddriver(), and they can be used to create sub-partition devices for MTD devices.

2. To save more space (equivalent to the size of one erase sector of MTD device) and to achieve faster read and write speeds, a method for direct writing was introduced at the FTL layer. This can be accomplished simply by using the following oflags during the open operation:

   • O_DIRECT. when this flag is passed in, ftl internally uses the direct write strategy and no read cache is used in ftl;
     otherwise, each write will be executed with the minimum granularity of flash erase sector size which means a
     "sector read back - erase sector - write sector" operation is performed by using a read cache buffer in heap.

   • O_SYNC. When this flag is passed in, we assume that the flash has been erased in advance and no erase operation
     will be performed internally within ftl. O_SYNC will take effect only when both O_DIRECT and O_SYNC are passed in
     simultaneously.

3. For uniformity, we remapped the mount flag in mount.h and unified it with the open flag in fcntl.h. The repetitive
   parts of their definitions were reused, and the remaining part of the mount flag redefine to the unused bit of open
   flags.**

Impact

1. The ftl_initialize() and bchdev_register() methods, originally used to wrap MTD device inodes, are no longer recommended for creating MTD device nodes. Instead, developers are advised to use the register_mtddriver(), register_partition_with_mtd(), or register_mtdpartition() functions. If these older methods were previously used in your code, please update your implementation to follow the new recommendations.

2. When an application wants to bypass the following logic in the FTL layer during the process of writing to Flash: using the sector size as the buffer for "read-back - erase - write", the MTD device node can be opened with the O_DIRECT flag. At this point, the FTL will directly write to the MTD device. The open method is as follows:
   fd = open("/dev/mtd_device", O_RDWR | O_DIRECT);

3. If the application ensures that the flash has been erased in advance, then adding the O_SYNC logic can control the FTL not to perform erasure. The open method is as follows:
   fd = open("/dev/mtd_device", O_RDWR | O_DIRECT | O_SYNC);

[xiaoxiang781216]

please remove the merge patch, @jingfei195887

[jerpelea]

please change commit title to
drivers/bch: Add direct writing method for FTL

thanks for adding the commit message

[michallenc]

Tested with example application in #16624, nxboot update and custom logging library.

[cederom]

Thank you @jingfei195887 looks interesting :-)

• @michallenc is out expert in this area so I follow all his remarks :-)
• What happens if we use SYNC flag assuming the memory is erased but it is not erased? Is this scenario handled properly with returning error code?
• Please update commits topic prefix to bch: ... and fs/mtd: ....
• Should we mark this change with [BREAKING] tag in the commit topic too? It changes filesystem / mount / mtd in a non-backward compatible way and this should be clearly marked if someone bumps the release and gets a broken code it will help find a fix quickly. Maybe it would be possible to create new function calls with additional flags as new functionality while old API remains untouched (could call new functions with default parameters)? What other folks think? :-)
• Should we also update the documentation on the new behavior (i.e. https://nuttx.apache.org/docs/latest/components/drivers/special/mtd.html)?

[TimJTi]

@michallenc @acassis @cederom @TimJTi this pr is ready for merging now, please review again.

@xiaoxiang781216 @jingfei195887 It's today's (additional!) task.

FYI - when I set up the test environment yesterday my existing code seemed to work OK without me yet having changed to the new methodology. So it's not a "breaking" change thankfully and I see the documentation recommends the new method, not demands it. Thank you :-)

[TimJTi]

this pr is ready for merging now, please review again.

@xiaoxiang781216 @jingfei195887 I have just tested this on my board, using NXboot to flash a recovery image from /dev/mtd0 to /dev/mtd2 and an "ota" update image from /dev/mtd1 to /dev/mtd0 having changed my startup code to directly register /dev/mtd%d instead of using FTL and BCH steps/

I can confirm it works for me.

If you want to invite me as a reviewer I'm happy to approve it: if not, I am equally happy to see it merged as soon as you're both done here.

I very much appreciate being kept in the loop and being given the opportunity to check it out before it just happened 👍 👍

[jingfei195887]

this pr is ready for merging now, please review again.

@xiaoxiang781216 @jingfei195887 I have just tested this on my board, using NXboot to flash a recovery image from /dev/mtd0 to /dev/mtd2 and an "ota" update image from /dev/mtd1 to /dev/mtd0 having changed my startup code to directly register /dev/mtd%d instead of using FTL and BCH steps/

I can confirm it works for me.

If you want to invite me as a reviewer I'm happy to approve it: if not, I am equally happy to see it merged as soon as you're both done here.

I very much appreciate being kept in the loop and being given the opportunity to check it out before it just happened 👍 👍

@TimJTi Thank you very much for helping test this feature and really appreciate your work!

[jingfei195887]

Hi @xiaoxiang781216 there is a CI build error in xtensa-01 for esp32-devkitc/nxlooper ，so I tried to do build in my own environment:

1. ./tools/refresh.sh --silent esp32-devkitc/nxloop
2. make

but another problem happened and I can't finish the build (even without my patch): maybe this is for a wrong version of xtensa gcc

does anybody konw why this is happening in xtensa01:

[xiaoxiang781216]

@tmedicci @eren-terzioglu could you look at the above problem?

[xiaoxiang781216]

let's ignore esp_tool ci error since it's unrelated to the change in this pr.

fixed

[fdcavalcanti]

Hi @xiaoxiang781216 there is a CI build error in xtensa-01 for esp32-devkitc/nxlooper ，so I tried to do build in my own environment:

1. ./tools/refresh.sh --silent esp32-devkitc/nxloop

2. make

but another problem happened and I can't finish the build (even without my patch): maybe this is for a wrong version of xtensa gcc

does anybody konw why this is happening in xtensa01:

Hi all, seems we still have this issue. I have run a git bisect since this CI error is still happening.
Looks like it was indeed introduced by drivers/fs: Control the behavior of FTL by passing oflags during the open process..
I'll be taking a look into it, but if you guys have any idea let me know.

[xiaoxiang781216]

it's strange why host esptool depend on the code change inside fs/mtd:(, @fdcavalcanti .

[michallenc]

@xiaoxiang781216 @jingfei195887 Hi guys, I have now tested the patch in my environment and all seems to work as expected. Thanks for all the work and effort with fixing the issues!

And thanks to @TimJTi for the quick review with nxboot.

[jingfei195887]

@xiaoxiang781216 @jingfei195887 Hi guys, I have now tested the patch in my environment and all seems to work as expected. Thanks for all the work and effort with fixing the issues!

And thanks to @TimJTi for the quick review with nxboot.

thankyou

[jingfei195887]

Hi @xiaoxiang781216 there is a CI build error in xtensa-01 for esp32-devkitc/nxlooper ，so I tried to do build in my own environment:

1. ./tools/refresh.sh --silent esp32-devkitc/nxloop

2. make

but another problem happened and I can't finish the build (even without my patch): maybe this is for a wrong version of xtensa gcc
does anybody konw why this is happening in xtensa01:

Hi all, seems we still have this issue. I have run a git bisect since this CI error is still happening. Looks like it was indeed introduced by drivers/fs: Control the behavior of FTL by passing oflags during the open process.. I'll be taking a look into it, but if you guys have any idea let me know.

Thanks for the helping, any process please let me know~

[fdcavalcanti]

Hi @xiaoxiang781216 there is a CI build error in xtensa-01 for esp32-devkitc/nxlooper ，so I tried to do build in my own environment:

1. ./tools/refresh.sh --silent esp32-devkitc/nxloop

2. make

but another problem happened and I can't finish the build (even without my patch): maybe this is for a wrong version of xtensa gcc
does anybody konw why this is happening in xtensa01:

Hi all, seems we still have this issue. I have run a git bisect since this CI error is still happening. Looks like it was indeed introduced by drivers/fs: Control the behavior of FTL by passing oflags during the open process.. I'll be taking a look into it, but if you guys have any idea let me know.

Thanks for the helping, any process please let me know~

So the issue is not related to this MR specifically. Its an alignment issue on ESP32 and it was a coincidence that happened on this PR. Upgrading to esptool v5.0.x fixes the problem and we should keep discussing on #16723.