Mikroe-STM32F4 port

[talkingtoken]

Hey Greg,

Being new to this forum, I'm not sure how much traffic you want to see regarding development and porting questions and/or suggestions, so if I'm posting inappropriately, just let me know. So I have made some more progress on the Mikroe-STM32F4 board port. I have the debug RS-232 port running (using the board's expansion header with jumper wires to an external USB-to-232 board). Plus I have the MMCSD interface working, and a driver for accessing the serial flash.

I have a few questions since I have and am making some fairly significant additions:

1. Are you interested in receiving this port for incorporation into a released build at some point? The questions below assume you might want the port, otherwise it wouldn't really matter how I implemented this stuff.

2. If you are interested in the port, in what format would you want it? Patch file? Or do you use a git repository or cvs or something?

3. I am adding a block flash driver that is optimized for the M25P80 (8Mbit serial flash) on this board (and other small serial flash), along with an optimized file system. I'm calling it the Sector Mapped Allocation for Really Tiny (SMART) flash driver and FS. But I need to add some FS ioctls. Should these be added to the "include/nuttx/fs/ioctl.h" or should I just put them in a local "driver/mtd/smart.h" file?

4. I plan to extend the file system's g_bdfsmap[] with a "smart" entry (properly conditioned with appropriate CONFIG_FS_SMART) so the serial flash can be mounted. Is this okay?

5. I noticed there are CONFIG_DEV_NULL and CONFIG_DEV_ZERO options, but the code doesn't actually use them. In my local "arch/arm/src/common/up_initialize.c" file, I conditioned the "devnull_register" with this define, and added a qualified devzero_register also, but I don't know if this would break anything in a release build.

6. I will need to add an enumeration to spi_dev_e in "include/nuttx/spi.h" for SPIDEV_AUDIO. I'm guessing that won't be an issue?

Again, sorry for all the pesky little questions. If there is a better forum for development type questions, just let me know.

Thanks. Oh, and did I mention this OS is FANTASTIC!!! I see there have been contributions from others, but it looks like more than 95% of the work is yours. Hard to believe one person could type so much! What keeps you so motivated?

Ken

[Gregory N]

Hi, Ken,

Post by talkingtoken
Being new to this forum, I'm not sure how much traffic you want to see regarding development and porting questions and/or suggestions, so if I'm posting inappropriately, just let me know. So I have made some more progress on the Mikroe-STM32F4 board port. I have the debug RS-232 port running (using the board's expansion header with jumper wires to an external USB-to-232 board). Plus I have the MMCSD interface working, and a driver for accessing the serial flash.

I don't have any problem with the email or answering questions. I presume that other people feel the same way, or they would be members of the list. Sometimes the level of email can get very high. But it has been quiet for some weeks now.

Post by talkingtoken
1. Are you interested in receiving this port for incorporation into a released build at some point? The questions below assume you might want the port, otherwise it wouldn't really matter how I implemented this stuff.

Yes. I would like to provide ports for any generally accessible boards. So it you want to contribute the code, I can help you get it incorporated.

Post by talkingtoken
2. If you are interested in the port, in what format would you want it? Patch file? Or do you use a git repository or cvs or something?

I use a GIT repository here: http://sourceforge.net/p/nuttx/git/ci/master/tree/

But patches will work fine. You can give patches to me at anytime; you don't have to wait until you are completely finished. A warning: I will make sure that files conform to the not-well-documented coding standards.

Post by talkingtoken
3. I am adding a block flash driver that is optimized for the M25P80 (8Mbit serial flash) on this board (and other small serial flash), along with an optimized file system. I'm calling it the Sector Mapped Allocation for Really Tiny (SMART) flash driver and FS. ...

I presume that this is a different driver that drivers/mtd/m25px.c. That driver currently supports M25P1, M25P32, M25P64, M25P128. I would think that adding the M25P80 to that driver would be the natural thing to do.

I am not sure how what you describe here is different. In most mounted FLASH file systems, there is a layering like this:

1) The Virtual File System (VFS) layer provides the interface between the user and the mounted file system
2) File system provides the interface to a block driver.
3) drivers/mtd/ftl.c provides the block driver interface to a Memory Technology Device (MTD) driver,
4) MTD drivers like, m25px.c, provide the low level flash/MTD interface

NXFFS, the NuttX wear-leveling FLASH file system, breaks this layering because it interfaces directly to the MTD driver.

It sounds like your SMART flash driver is a combination of 2 through 4. Am I understanding that right? I can see that you might be able to do some optimizations by combining those layers.

Post by talkingtoken
... But I need to add some FS ioctls. Should these be added to the "include/nuttx/fs/ioctl.h" or should I just put them in a local "driver/mtd/smart.h" file?

Ultimately, from the end user's point of view, they open a file (or a character driver) then call ioctl() against the file/drivers' file descriptor. So it does not seem like the user should have to know what the underlying drivers are to use the ioctl, should they? That argument suggests that include/nuttx/fs/ioctl.h would be the correct place.

There is a precedence for using include/nuttx/fs/ioctl.h. There are existing drivers specific ioctl commands using the FIOC, BIOC, and MTDIOC macros in include/nuttx/fs/ioctl.h. Look at, for example, FIOC_MMAP, BIOC_XIPBASE, and MTDIOC_XIPBASE. These are all closely related IOCTLs to support XiP.

Post by talkingtoken
4. I plan to extend the file system's g_bdfsmap[] with a "smart" entry (properly conditioned with appropriate CONFIG_FS_SMART) so the serial flash can be mounted. Is this okay?

Certainly. That is a part of the OS that I planned to re-design some day in my spare time. Currently, you do have to hard-code file systems in these tables. I would prefer to see a registration interface so that we can dynamically add file systems. Someday.

Post by talkingtoken
5. I noticed there are CONFIG_DEV_NULL and CONFIG_DEV_ZERO options, but the code doesn't actually use them. In my local "arch/arm/src/common/up_initialize.c" file, I conditioned the "devnull_register" with this define, and added a qualified devzero_register also, but I don't know if this would break anything in a release build.

No, that is a correct change.

Post by talkingtoken
6. I will need to add an enumeration to spi_dev_e in "include/nuttx/spi.h" for SPIDEV_AUDIO. I'm guessing that won't be an issue?

No problem.

Post by talkingtoken
Again, sorry for all the pesky little questions. If there is a better forum for development type questions, just let me know.
Thanks. Oh, and did I mention this OS is FANTASTIC!!! I see there have been contributions from others, but it looks like more than 95% of the work is yours. Hard to believe one person could type so much! What keeps you so motivated?

I have put a lot of effort into it. I was looking at Ohloh a few days ago it claims that there is 1.2 million lines of code in the repository. And, yes, most of that is my work. Other people have made significant contributions; I certainly don't want to minimize that. But I am the only person who has been working on this day-in/day-out.

If anyone else is interested in being a developer on the project, that would be great! But so far, I am the only person who has made that commitment.

Greg

[talkingtoken]

Hi Greg,

Post by Gregory N

Post by talkingtoken
1. Are you interested in receiving this port for incorporation into a released build at some point? The questions below assume you might want the port, otherwise it wouldn't really matter how I implemented this stuff.

Yes. I would like to provide ports for any generally accessible boards. So it you want to contribute the code, I can help you get it incorporated.

Excellent, I'm a big proponent of open source development. I have an emulator for the TRS-80 Model 100 portable (circa 1983) that I have been working on since 2004. It is hosted on sourceforge also: http://souceforge.net/projects/virtualt

Post by Gregory N
I use a GIT repository here: http://sourceforge.net/p/nuttx/git/ci/master/tree/
But patches will work fine. You can give patches to me at anytime; you don't have to wait until you are completely finished. A warning: I will make sure that files conform to the not-well-documented coding standards.

I can submit patch files until you feel comfortable making me a developer. How should I submit them? Posts to this forum? Direct email?

Post by Gregory N

Post by talkingtoken
3. I am adding a block flash driver that is optimized for the M25P80 (8Mbit serial flash) on this board (and other small serial flash), along with an optimized file system. I'm calling it the Sector Mapped Allocation for Really Tiny (SMART) flash driver and FS. ...

I presume that this is a different driver that drivers/mtd/m25px.c. That driver currently supports M25P1, M25P32, M25P64, M25P128. I would think that adding the M25P80 to that driver would be the natural thing to do.

Yes, I have extended the m25px.c file to support the M25P80 device on this board. This part reports manufacturer and memory type ID bytes that I have not found in any other driver (it's an EN25F80 by Eon Silicon). If follows standard M25P80 commands with a block erase size of 64K, but also has a "sector" erase size of 4K (command 0x20). So far, I have only been using the more widely available block erase and haven't even added the sector erase command to the code.

Post by Gregory N
1) The Virtual File System (VFS) layer provides the interface between the user and the mounted file system
2) File system provides the interface to a block driver.
3) drivers/mtd/ftl.c provides the block driver interface to a Memory Technology Device (MTD) driver,
4) MTD drivers like, m25px.c, provide the low level flash/MTD interface
NXFFS, the NuttX wear-leveling FLASH file system, breaks this layering because it interfaces directly to the MTD driver.
It sounds like your SMART flash driver is a combination of 2 through 4. Am I understanding that right? I can see that you might be able to do some optimizations by combining those layers.

What I'm working on is at layers 2 and 3. I looked at the nxffs and ftl drivers, but they both want to allocate a buffer the size of a typical erase block (64K), and keep it around. The SMART block/ftl driver (layer 2) only needs to allocate 4-6K (base on selected sector size), so it is optimized for minimal RAM, sacrificing performance. It provides standard read / write capability for usual block operations for use with 'dd', etc.

But this isn't really the best access method because it deals with blocks vs. SMART sectors, thus the reason I want to add some SMART specific ioctls. These would provide the sector managed interface needed by the SMART FS driver, which would provide the layer 3 functionality. I suppose I could do like the NXFFS driver does and combine these two layers, then talk directly with the MTD device, but I like having the /dev/mtdsmart0 block device for testing. Plus it would be useful for anyone who wanted to use the flash without an FS, such as for holding images, resources, etc.

Post by Gregory N

Post by talkingtoken
... But I need to add some FS ioctls. Should these be added to the "include/nuttx/fs/ioctl.h" or should I just put them in a local "driver/mtd/smart.h" file?

Ultimately, from the end user's point of view, they open a file (or a character driver) then call ioctl() against the file/drivers' file descriptor. So it does not seem like the user should have to know what the underlying drivers are to use the ioctl, should they? That argument suggests that include/nuttx/fs/ioctl.h would be the correct place.
There is a precedence for using include/nuttx/fs/ioctl.h. There are existing drivers specific ioctl commands using the FIOC, BIOC, and MTDIOC macros in include/nuttx/fs/ioctl.h. Look at, for example, FIOC_MMAP, BIOC_XIPBASE, and MTDIOC_XIPBASE. These are all closely related IOCTLs to support XiP.

Okay, I'll plan to put them there for now -- they can always be moved. They are only needed between the SMART block driver and its FS driver, not by the end user app. The smart FS driver will just be a standard FS interface. It will provide only limited wear-leveling, so it really isn't design for writing to on a daily basis ... more like holding configuration info and things that stay somewhat static.

Post by Gregory N

Post by talkingtoken
5. I noticed there are CONFIG_DEV_NULL and CONFIG_DEV_ZERO options, but the code doesn't actually use them. In my local "arch/arm/src/common/up_initialize.c" file, I conditioned the "devnull_register" with this define, and added a qualified devzero_register also, but I don't know if this would break anything in a release build.

No, that is a correct change.

Cool. Perhaps that can be the 1st patch file I submit. It's really simple.

Post by Gregory N
If anyone else is interested in being a developer on the project, that would be great! But so far, I am the only person who has made that commitment.

I just might take you up on that! Time is always the limiting factor, especially with a 1.5 yr old daughter :)

Ken

[Gregory N]

Hi, Ken,

Post by talkingtoken
Excellent, I'm a big proponent of open source development. I have an emulator for the TRS-80 Model 100 portable (circa 1983) that I have been working on since 2004. It is hosted on sourceforge also: http://souceforge.net/projects/virtualt

Now you are talking about one of my favorite things; retro computing. The Model 100 was an 8085 right? My first computer was a model 1 and my first job was programming 8085s.

There is a port of NuttX the the Z80 based TRS-80s using the XTRs simulation. You can see that in the source tree at configs/xtrs. I doubt that has been used in years and so probably has some bit rot.

And I am still waiting for my Kickstarter P112 hardware (Z180).

Post by talkingtoken
I can submit patch files until you feel comfortable making me a developer. How should I submit them? Posts to this forum? Direct email?

Directly to me is probably better.

Post by talkingtoken

Post by Gregory N
1) The Virtual File System (VFS) layer provides the interface between the user and the mounted file system
2) File system provides the interface to a block driver.
3) drivers/mtd/ftl.c provides the block driver interface to a Memory Technology Device (MTD) driver,
4) MTD drivers like, m25px.c, provide the low level flash/MTD interface
...
It sounds like your SMART flash driver is a combination of 2 through 4. Am I understanding that right? I can see that you might be able to do some optimizations by combining those layers.

What I'm working on is at layers 2 and 3. I looked at the nxffs and ftl drivers, but they both want to allocate a buffer the size of a typical erase block (64K), and keep it around. The SMART block/ftl driver (layer 2) only needs to allocate 4-6K (base on selected sector size), so it is optimized for minimal RAM, sacrificing performance. It provides standard read / write capability for usual block operations for use with 'dd', etc.

Most FLASH parts will have erase-able unit (sectors, page, whatever) that is smaller that the fully erase block. More like 8KB. How do a read-modify-write without buffering the erase block? You must do some interesting juggling.

Post by talkingtoken

Post by Gregory N

Post by talkingtoken
... But I need to add some FS ioctls. Should these be added to the "include/nuttx/fs/ioctl.h" or should I just put them in a local "driver/mtd/smart.h" file?

...
There is a precedence for using include/nuttx/fs/ioctl.h. There are existing drivers specific ioctl commands using the FIOC, BIOC, and MTDIOC macros in include/nuttx/fs/ioctl.h. Look at, for example, FIOC_MMAP, BIOC_XIPBASE, and MTDIOC_XIPBASE. These are all closely related IOCTLs to support XiP.

Okay, I'll plan to put them there for now -- they can always be moved. They are only needed between the SMART block driver and its FS driver, not by the end user app. The smart FS driver will just be a standard FS interface. It will provide only limited wear-leveling, so it really isn't design for writing to on a daily basis ... more like holding configuration info and things that stay somewhat static.

The same is true of some of the other IOCTLs in that file as well.

Greg

[Michael Smith]

What I'm working on is at layers 2 and 3. I looked at the nxffs and ftl drivers, but they both want to allocate a buffer the size of a typical erase block (64K), and keep it around. The SMART block/ftl driver (layer 2) only needs to allocate 4-6K (base on selected sector size), so it is optimized for minimal RAM, sacrificing performance. It provides standard read / write capability for usual block operations for use with 'dd', etc.

Most FLASH parts will have erase-able unit (sectors, page, whatever) that is smaller that the fully erase block. More like 8KB. How do a read-modify-write without buffering the erase block? You must do some interesting juggling.

Actually, lots of flash parts don't (e.g. NAND flash). The usual technique is to move-and-compact; you only need a pool of IO-sized buffers to do a streaming copy, and obviously a portion of the overall device reserved for sparing, buffering, garbage-collection transients, etc.

= Mike

[talkingtoken]

Post by Michael Smith

What I'm working on is at layers 2 and 3. I looked at the nxffs and ftl drivers, but they both want to allocate a buffer the size of a typical erase block (64K), and keep it around. The SMART block/ftl driver (layer 2) only needs to allocate 4-6K (base on selected sector size), so it is optimized for minimal RAM, sacrificing performance. It provides standard read / write capability for usual block operations for use with 'dd', etc.

Most FLASH parts will have erase-able unit (sectors, page, whatever) that is smaller that the fully erase block. More like 8KB. How do a read-modify-write without buffering the erase block? You must do some interesting juggling.

Actually, lots of flash parts don't (e.g. NAND flash). The usual technique is to move-and-compact; you only need a pool of IO-sized buffers to do a streaming copy, and obviously a portion of the overall device reserved for sparing, buffering, garbage-collection transients, etc.

Yep, that's the technique I'm implementing. Each physical sector has a virtual sector number associated with it and can be "released" for garbage collection. Files (and probably directories) consist of a chain of logical sector numbers, so if a file's sectors get moved, it doesn't matter because the virtual to physical mapping gets updated and everything stays linked (at least that's the plan). But it does require reserving a couple of spare erase blocks (or depending on an SD card sitting in an MMC/SD slot somewhere, which would be REALLY nasty, though it would work).

Ken

[Gregory N]

Hi, Mike,

Post by Michael Smith

What I'm working on is at layers 2 and 3. I looked at the nxffs and ftl drivers, but they both want to allocate a buffer the size of a typical erase block (64K), and keep it around. The SMART block/ftl driver (layer 2) only needs to allocate 4-6K (base on selected sector size), so it is optimized for minimal RAM, sacrificing performance. It provides standard read / write capability for usual block operations for use with 'dd', etc.

Most FLASH parts will have erase-able unit (sectors, page, whatever) that is smaller that the fully erase block. More like 8KB. How do a read-modify-write without buffering the erase block? You must do some interesting juggling.

Actually, lots of flash parts don't (e.g. NAND flash). The usual technique is to move-and-compact; you only need a pool of IO-sized buffers to do a streaming copy, and obviously a portion of the overall device reserved for sparing, buffering, garbage-collection transients, etc.

There is no NAND support in the tree now. Most FLASH drivers have to address the wear leveling issue. But with NAND you have to address sparing and ECC.

Greg

[Gregory N]

A little more on this topic...

Post by talkingtoken
What I'm working on is at layers 2 and 3. I looked at the nxffs and ftl drivers, but they both want to allocate a buffer the size of a typical erase block (64K), and keep it around. The SMART block/ftl driver (layer 2) only needs to allocate 4-6K (base on selected sector size), so it is optimized for minimal RAM, sacrificing performance. It provides standard read / write capability for usual block operations for use with 'dd', etc.

It has been awhile since I have looked at or thought very much about NXFFS. But as I recall that but buffer is only used when FLASH memory is full and NXFFS needs to reorganize it to regain space. So it is probably not necessary and NXFFS could be optimized to eliminate it.

There is one thing that I dislike about NXFFS: Like JFFS2 and other wear leveling FLASH file systems, it takes a significant amount of time for the file system to become ready after power on. The entire FLASH needs to be scanned.

That could be done on a background thread and might not necessarily be an issue. But currently, when the FLASH needs to be reorganized, then NXFSS can stall for quite awhile.

So another thing that NXFFS needs is a daemon like the one used with JFFS2 that constantly collects garbage so that there is never any need for a wholesale reorganization of FLASH.

It sounds like your SMART file system addresses all of these issues. That would be a welcome addition.

Greg

[talkingtoken]

Post by Gregory N
A little more on this topic...
It has been awhile since I have looked at or thought very much about NXFFS. But as I recall that but buffer is only used when FLASH memory is full and NXFFS needs to reorganize it to regain space. So it is probably not necessary and NXFFS could be optimized to eliminate it.
There is one thing that I dislike about NXFFS: Like JFFS2 and other wear leveling FLASH file systems, it takes a significant amount of time for the file system to become ready after power on. The entire FLASH needs to be scanned.
That could be done on a background thread and might not necessarily be an issue. But currently, when the FLASH needs to be reorganized, then NXFSS can stall for quite awhile.
So another thing that NXFFS needs is a daemon like the one used with JFFS2 that constantly collects garbage so that there is never any need for a wholesale reorganization of FLASH.
It sounds like your SMART file system addresses all of these issues. That would be a welcome addition.

The SMART file system would have to perform a quick scan of the flash during mount also, but it would be limited to 5 bytes per logical sector. So the larger you make your logical sectors, the faster it will mount (also reduces per-sector management overhead, however it increases the granularity for allocation, so it's a tradeoff).

Let's hope the implementation can live up to its name, and that I didn't totally and completely give it the wrong FLA (five letter acronym, but usually referred to as TLA).

Ken

[Michael Smith]

Post by talkingtoken
Being new to this forum, I'm not sure how much traffic you want to see regarding development and porting questions and/or suggestions, so if I'm posting inappropriately, just let me know.

Speaking as a not-Greg subscriber to this list, I'm here expressly to see and (if I'm lucky, participate in) these sorts of discussions. So if you'll take a vote from the peanut gallery, please do continue to share your questions and observations. 8)

= Mike

[talkingtoken]

Post by Michael Smith

Post by talkingtoken
Being new to this forum, I'm not sure how much traffic you want to see regarding development and porting questions and/or suggestions, so if I'm posting inappropriately, just let me know.

Speaking as a not-Greg subscriber to this list, I'm here expressly to see and (if I'm lucky, participate in) these sorts of discussions. So if you'll take a vote from the peanut gallery, please do continue to share your questions and observations. 8)

Hey Mike,

Sorry, didn't see your post earlier .. I was viewing them from my email account, and the message thread logic on gmail buried your post.

Votes from peanut galleries are always welcomed as far as I'm concerned! In fact, I'm usually in the gallery along with everyone else.

Ken