Discussion:
Problem with commit from Juha Niskanen committed 4f18b40,2017-05-10
(too old to reply)
Pascal Speck iktek01@yahoo.com [nuttx]
2018-02-14 12:36:36 UTC
Permalink
Hi guys,

At 2017-05-10 Juha commited a change that adds an MTD_ERASE to byte write function.

Within this the config does not work any more for me, because I have a single eraseblock per partition and an erase of single parts of that is not possible.

I understand the config partition to not having to erase anything as long as new blocks are available, so I can't understand the intend of this commit.

Regards Pascal
'Juha Niskanen (Haltian)' juha.niskanen@haltian.com [nuttx]
2018-02-15 14:59:39 UTC
Permalink
(I don't have easy access to NuttX sources from my current location/computer, so writing this from memory)


"At 2017-05-10 Juha commited a change that adds an MTD_ERASE to byte write function."


I don't think so. mtdconfig_writebytes() is not a byte write function as it implements both block and byte writes. I recall adding MTD_ERASE between block read and write that updates block's contents in-place.


"I understand the config partition to not having to erase anything as long as new blocks are available,"


The need to erase a block before writes that turn 0 bits back to 1 bits is a fundamental property of flash memory. The config partition updates (at least some) data structures in-place, not appending new blocks to an immutable log. The latter approach would needlessly limit the number of config updates if could not reclaim space from accumulated stale blocks.


I used MTD config in top of MTD progmem and MTD partition so there is no FTL layer to hide the true nature of underlying flash. If you want "write once, read many" semantics you could have a lower layer with dummy erase-method or use byte writes only.


Best Regards,

Juha

________________________________
From: ***@yahoogroups.com <***@yahoogroups.com> on behalf of Pascal Speck ***@yahoo.com [nuttx] <***@yahoogroups.com>
Sent: Wednesday, February 14, 2018 2:36:36 PM
To: ***@yahoogroups.com
Subject: [nuttx] Problem with commit from Juha Niskanen committed 4f18b40,2017-05-10



Hi guys,

At 2017-05-10 Juha commited a change that adds an MTD_ERASE to byte write function.

Within this the config does not work any more for me, because I have a single eraseblock per partition and an erase of single parts of that is not possible.

I understand the config partition to not having to erase anything as long as new blocks are available, so I can't understand the intend of this commit..

Regards Pascal
Pascal Speck iktek01@yahoo.com [nuttx]
2018-02-16 08:06:18 UTC
Permalink
Hi Juha,

unfortunately your implementation does not work for all flash chips. I have a chip with a big erase size and much lower block size.
Somehow your implementation mixes that up, because your implementation assumes that a block can be erased individually which is not true for most flash chips as there is a much bigger erase-size.
Post by 'Juha Niskanen (Haltian)' ***@haltian.com [nuttx]
The need to erase a block before writes that turn 0 bits back to 1 bits is a fundamental property of flash memory. The config partition updates (at least some) data structures in-place, not
appending new blocks to an immutable log. The latter approach would� <
Post by 'Juha Niskanen (Haltian)' ***@haltian.com [nuttx]
needlessly limit the number of config updates if could not reclaim space from accumulated stale blocks.
You're right, but this is exactly what the "mtdconfig_consolidate" or "mtdconfig_ramconsolidate" function should do whenever there are no more config items available.

So the approach of the driver is the following:

1. Config Partition is formatted and cleared -> All Config blocks are set to 0xFF and can be written.
2. Some config blocks are written (phdr->id is changed to a valid ID ( not 0 not 0xff )
3. When a config item is updated the old one is set to 0 (invalid)
��� if a new one is available, we set that id to the valid id.
-> until this point it would be as you mentioned... but:

If none is available we try to consolidate (either via ram if we have only one eraseblock or with the help of the other erase-page). Within this consolidation the whole erase-block will be cleared and
only the "active" items will be written back to the config-partition again.

So there is no need to erase something in the flash when a config item is updated and I think this was then indent of writing it that way in the past.

Maybe there has been another bug for that your driver did try to write to an "uncleared" ram-area, but clearing within the block-write function is not easily possible because you would clear the whole
"eraseblock" which might include other blocks, too.

I hope you could follow my thoughts...

Regards Pascal
Post by 'Juha Niskanen (Haltian)' ***@haltian.com [nuttx]
�
(I don't have easy access to NuttX sources from my current location/computer, so writing this from memory)
"At 2017-05-10 Juha commited a change that adds an MTD_ERASE to byte write function."
I don't think so. mtdconfig_writebytes() is not a byte write function as it implements both block and byte writes. I recall adding MTD_ERASE between block read and write that updates block's
contents in-place.
"I understand the config partition to not having to erase anything as long as new blocks are available,"
The need to erase a block before writes that turn 0 bits back to 1 bits is a fundamental property of flash memory. The config partition updates (at least some) data structures in-place, not
appending new blocks to an immutable log. The latter approach would needlessly limit the number of config updates if could not reclaim space from accumulated stale blocks.
I used MTD config in top of MTD progmem and MTD partition so there is no FTL layer to hide the true nature of underlying flash. If you want "write once, read many" semantics you could have a lower
layer with dummy erase-method or use byte writes only.
Best Regards,
�� Juha
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*Sent:* Wednesday, February 14, 2018 2:36:36 PM
*Subject:* [nuttx] Problem with commit from Juha Niskanen committed 4f18b40,2017-05-10
�
�
Hi guys,
At 2017-05-10 Juha commited a change that adds an MTD_ERASE to byte write function.
Within this the config does not work any more for me, because I have a single eraseblock per partition and an erase of single parts of that is not possible.
I understand the config partition to not having to erase anything as long as new blocks are available, so I can't understand the intend of this commit.
Regards Pascal
------------------------------------

------------------------------------


------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/nuttx/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/nuttx/join
(Yahoo! ID required)

<*> To change settings via email:
nuttx-***@yahoogroups.com
nuttx-***@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
nuttx-***@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
https://info.yahoo.com/legal/us/yahoo/utos/terms/
'Juha Niskanen (Haltian)' juha.niskanen@haltian.com [nuttx]
2018-02-19 12:47:09 UTC
Permalink
Hi Pascal & list,


Now I see it.... the underlying root cause is that some flash implementation, notably MCU progmem, do not behave like (most?) external flash chips and the mtd_progmem does not handle this. There are some ways to fix this:


1. read-erase-write full erase block instead of data block. This works with oddball drivers but causes unnecessary performance problems for others (big partitions with few huge erase blocks) and works only for /dev/config
2. Augment progmem API to export information about MCU flash quirks like alignment restrictions (EFM32 requires 32-bit access, STM32L4 64-bit, SAMv71 128-bit) and silicon design flaws (STM32L4 does not allow idempotent "1" -> "1" writes without erasing entire page) and make mtd-progmem layer work around these.
3. Make sure each and every flash driver behaves as assumed by typical flash filesystems and other upper layer code.


I have rewritten the STM32L4 flash driver to confirm with 3), that is, it now allows arbitrary byte writes with arbitrary alignment as expected by config & mtd-progmem layers. Since MTD-progmem can do multiple writes per page, the chip layer uses a page buffer like SAM progmem. Patches are forthcoming.


Best Regards,

Juha


________________________________
From: ***@yahoogroups.com <***@yahoogroups.com> on behalf of Pascal Speck ***@yahoo.com [nuttx] <***@yahoogroups.com>
Sent: Friday, February 16, 2018 10:06 AM
To: ***@yahoogroups.com
Subject: Re: [nuttx] Problem with commit from Juha Niskanen committed 4f18b40,2017-05-10

Hi Juha,

unfortunately your implementation does not work for all flash chips. I have a chip with a big erase size and much lower block size.
Somehow your implementation mixes that up, because your implementation assumes that a block can be erased individually which is not true for most flash chips as there is a much bigger erase-size.
Post by 'Juha Niskanen (Haltian)' ***@haltian.com [nuttx]
The need to erase a block before writes that turn 0 bits back to 1 bits is a fundamental property of flash memory. The config partition updates (at least some) data structures in-place, not
appending new blocks to an immutable log. The latter approach wouldᅵ <
Post by 'Juha Niskanen (Haltian)' ***@haltian.com [nuttx]
needlessly limit the number of config updates if could not reclaim space from accumulated stale blocks.
You're right, but this is exactly what the "mtdconfig_consolidate" or "mtdconfig_ramconsolidate" function should do whenever there are no more config items available.

So the approach of the driver is the following:

1. Config Partition is formatted and cleared -> All Config blocks are set to 0xFF and can be written.
2. Some config blocks are written (phdr->id is changed to a valid ID ( not 0 not 0xff )
3. When a config item is updated the old one is set to 0 (invalid)
ᅵᅵᅵ if a new one is available, we set that id to the valid id.
-> until this point it would be as you mentioned... but:

If none is available we try to consolidate (either via ram if we have only one eraseblock or with the help of the other erase-page). Within this consolidation the whole erase-block will be cleared and
only the "active" items will be written back to the config-partition again.

So there is no need to erase something in the flash when a config item is updated and I think this was then indent of writing it that way in the past.

Maybe there has been another bug for that your driver did try to write to an "uncleared" ram-area, but clearing within the block-write function is not easily possible because you would clear the whole
"eraseblock" which might include other blocks, too.

I hope you could follow my thoughts...

Regards Pascal
Post by 'Juha Niskanen (Haltian)' ***@haltian.com [nuttx]
ᅵ
(I don't have easy access to NuttX sources from my current location/computer, so writing this from memory)
"At 2017-05-10 Juha commited a change that adds an MTD_ERASE to byte write function."
I don't think so. mtdconfig_writebytes() is not a byte write function as it implements both block and byte writes. I recall adding MTD_ERASE between block read and write that updates block's
contents in-place.
"I understand the config partition to not having to erase anything as long as new blocks are available,"
The need to erase a block before writes that turn 0 bits back to 1 bits is a fundamental property of flash memory. The config partition updates (at least some) data structures in-place, not
appending new blocks to an immutable log. The latter approach would needlessly limit the number of config updates if could not reclaim space from accumulated stale blocks.
I used MTD config in top of MTD progmem and MTD partition so there is no FTL layer to hide the true nature of underlying flash. If you want "write once, read many" semantics you could have a lower
layer with dummy erase-method or use byte writes only.
Best Regards,
ᅵᅵ Juha
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*Sent:* Wednesday, February 14, 2018 2:36:36 PM
*Subject:* [nuttx] Problem with commit from Juha Niskanen committed 4f18b40,2017-05-10
ᅵ
ᅵ
Hi guys,
At 2017-05-10 Juha commited a change that adds an MTD_ERASE to byte write function.
Within this the config does not work any more for me, because I have a single eraseblock per partition and an erase of single parts of that is not possible.
I understand the config partition to not having to erase anything as long as new blocks are available, so I can't understand the intend of this commit.
Regards Pascal
------------------------------------

------------------------------------


------------------------------------

Yahoo Groups Links

Loading...