changing partition types at run time

[pljakobs]

I have another question around partitions and OTA, hopefully not as confusing as the last one.

Currently, my partition map is like this:

Device            Start       End         Size        Type      SubType   Name              Filename
----------------  ----------  ----------  ----------  --------  --------  ----------------  ------------
spiFlash          0x00000000  0x00001fff          8K                      Boot Sector       
spiFlash          0x00002000  0x000f9fff        992K  app       ota_0     rom0              $(RBOOT_ROM_0_BIN)
spiFlash          0x000fa000  0x000fffff         24K                      (unused)          
spiFlash          0x00100000  0x001bffff        768K  data      spiffs    spiffs0           $(SPIFF_BIN_OUT)
spiFlash          0x001c0000  0x00201fff        264K                      (unused)          
spiFlash          0x00202000  0x002f9fff        992K  app       ota_1     rom1              $(RBOOT_ROM_0_BIN)
spiFlash          0x002fa000  0x002fffff         24K                      (unused)          
spiFlash          0x00300000  0x003bffff        768K  data      spiffs    spiffs1           $(SPIFF_BIN_OUT)
spiFlash          0x003c0000  0x003f9fff        232K                      (unused)          
spiFlash          0x003fa000  0x003fafff          4K                      Partition Table   
spiFlash          0x003fb000  0x003fbfff          4K  data      rfcal     rf_cal            
spiFlash          0x003fc000  0x003fcfff          4K  data      phy       phy_init          $(FLASH_INIT_DATA)
spiFlash          0x003fd000  0x003fffff         12K  data      sysparam  sys_param       

since I have now managed to get my web frontend small enough to fit into the rom partition, I want to change my data partitions. Currently, every OTA brings a new rom and spiffs partition, in future, the data partitions won't need to be part of the OTA process anymore.

What I'm planning is this:
I want to chage both data partitions into LittleFS as, as far as I could find out, it does better wear levelling than spiffs. (one issue we have had is that the controller writes one specific value sometimes a hundred times a day and wears out the flash within a year or two).
I will use the data partitions to store configration, but also things like presets, scenes, other controllers etc. All this data should persist through OTA, which it will easily do if I don't flash over it.
The critical part is the transition.

And there my question is:

can I change the type of a partition at run time?
I know that when talking about the OTA from Sming4.2 based controllers, there is code to write a partition table initially, but can I also change the existing partitions?

My process would be like this:

• create first LittleFS in the place where the 1st spiffs partition is, at the same time extending it to the 1MB boundry
• copy config data from remaining spiffs to LittleFS
• create the second LittleFS partitions

from then on, since I don't have use for more than 1MByte of storage, I'm thinking I'll keep the 2nd partition as a "last known good" backup and if I ever encounter a situation where I can't mount the first LittleFS, I use that and signal some degraded state to the user.

would the following code actually overwrite the existing partition table entry or would it create a conflicting one?

Storage::initialize();

auto& table = Storage::spiFlash->editablePartitions();
if(!table) {
using SubType = Storage::Partition::SubType;
table.add(F("spiffs0"), SubType::Data::littlefs, 0x200000, 0xc201fff);
}

and would that partition be immediately available to create a file system in?

I'm currently trying to build a minimal prototype on the base of Basic_Storage, but

	auto& table=Storage::spiFlash->editablePartitions();
	//using SubType=Storage::Partition::SubType;
	//table.add(F("littlefs0"), SubType::Data::littlefs, 0x40000, 0x5ffff));

I'm getting

/home/pjakobs/devel/Basic_Storage/app/application.cpp:73:23: error: 'spiFlash' is not a member of 'Storage'; did you mean 'SpiFlash'?
   73 |  auto& table=Storage::spiFlash->editablePartitions();
      |                       ^~~~~~~~
      |                       SpiFlash

[mikee47]

spiFlash isn't made available by default, use is discouraged but sometimes required. Add #include <Storage/SpiFlash.h>.

[mikee47]

can I change the type of a partition at run time? I know that when talking about the OTA from Sming4.2 based controllers, there is code to write a partition table initially, but can I also change the existing partitions?

A Partition object is designed to protect the actual partition entries from modification, but you can do this:

/*
  * Provide access to the protected `Partition::Info` structure.
  * Add methods as required to update entry information.
  */
class EditablePartition : public Storage::Partition
{
public:
	void setType(FullType fullType)
	{
		auto info = const_cast<Info*>(mPart);
		info->type = fullType.type;
		info->subtype = fullType.subtype;
	}

	void setName(const String& name)
	{
		auto info = const_cast<Info*>(mPart);
		info->name = name;
	}
};

void changeSpiffs0ToLittleFS()
{
	auto part = Storage::findPartition("spiffs0");
	if(part) {
		auto& ep = static_cast<EditablePartition&>(part);
		ep.setType(Storage::Partition::SubType::Data::littlefs);
		ep.setName(F("littlefs0"));
	}
}

would the following code actually overwrite the existing partition table entry or would it create a conflicting one?

Storage::initialize();

auto& table = Storage::spiFlash->editablePartitions();
if(!table) {
	using SubType = Storage::Partition::SubType;
	table.add(F("spiffs0"), SubType::Data::littlefs, 0x200000, 0xc201fff);
}

Because of if(!table) this would only add the entry to an empty partition table. Without that check it would create a conflicting entry, but because it's added at the end of the table the existing "spiffs0" partition is the one which would be found using Storage::findPartition("spiffs0").

[pljakobs]

I've been playing around with this for a while now, initially just adding a partition, and it seems to me that the change is only in RAM but not persisted to storage.

  spiFlash: type flash, size 0x400000. Partitions:
    spiFlash/rom0, app/factory @ 0x2000, size 0x80000
    spiFlash/user0, 64.0 @ 0x82000, size 0x4000
    spiFlash/user1, 64.1 @ 0xfe000, size 0x4000
    spiFlash/spiffs0, data/spiffs @ 0x308000, size 0x80000
    spiFlash/spiffs1, data/spiffs @ 0x388000, size 0x3c000
    spiFlash/spiffs2, data/spiffs @ 0x3c4000, size 0x36000
    spiFlash/rf_cal, data/rfCal @ 0x3fb000, size 0x1000
    spiFlash/phy_init, data/phy @ 0x3fc000, size 0x1000
    spiFlash/sys_param, data/sysParam @ 0x3fd000, size 0x3000


creating littlefs partition
size of partition table: 4
spiFlash/littlefs: 01 00 00 00 f0 0f ff f7 6c 69 74 74 6c 65 66 73  ........littlefs
                   2f e0 00 10 00 00 02 00 00 10 00 00 02 00 00 00  /...............
                   ff 00 00 00 ff ff ff 7f fe 03 00 00 70 1f fc 08  ............p...
                   53 60 4b 42 ff ff ff ff ff ff ff ff ff ff ff ff  S`KB............
                   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
                   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
                   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
                   ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  ................
Elapsed: 1430us
Speed:   2864 KB/s

388064 heap used: 0, mount() returned 0 (Success)
list Devices 

Registered storage devices:
  spiFlash: type flash, size 0x400000. Partitions:
    spiFlash/rom0, app/factory @ 0x2000, size 0x80000
    spiFlash/user0, 64.0 @ 0x82000, size 0x4000
    spiFlash/user1, 64.1 @ 0xfe000, size 0x4000
    spiFlash/spiffs0, data/spiffs @ 0x308000, size 0x80000
    spiFlash/spiffs1, data/spiffs @ 0x388000, size 0x3c000
    spiFlash/spiffs2, data/spiffs @ 0x3c4000, size 0x36000
    spiFlash/rf_cal, data/rfCal @ 0x3fb000, size 0x1000
    spiFlash/phy_init, data/phy @ 0x3fc000, size 0x1000
    spiFlash/sys_param, data/sysParam @ 0x3fd000, size 0x3000
    spiFlash/littlefs, data/littlefs @ 0x200000, size 0x2000


reading partition table from spiFlash
printing littlefs again
Partition 'littlefs' not found

[mikee47]

I've been playing around with this for a while now, initially just adding a partition, and it seems to me that the change is only in RAM but not persisted to storage.

Yes. As you're "into the weeds" here I suggest reading the source code to gain a better understanding of what is happening. Sming calls Storage::initialize() right before init() in Sming/Components/Storage/src/Storage.cpp so that's a good place to start.

[mikee47]

NB. We've already covered updating the on-flash partition table in #2727

[pljakobs]

re already covered - that's where I started but as said can't see that actually writing to flash and also, it only allows to add partitions. My tests make me think that the code we discussed there (using editablePartitions) would not have created a partition table on the storage device.
I guess I'll try with the code in the 2nd example here
https://sming.readthedocs.io/en/latest/_inc/Sming/Components/Storage/ota-migration.html

[mikee47]

Here's my suggested approach to your particular situation. Code compiles but is otherwise untested. You'll need to add LittleFS to your project's COMPONENT_DEPENDS.

#include <Storage/SpiFlash.h>
#include <LittleFS.h>
#include <IFS/FileCopier.h>

enum class MigrateResult {
	failure,
	alreadyUpgraded,
	success,
};

MigrateResult migrateFilesystem()
{
	/*
	 * Locate the original SPIFFS partition we intend to upgrade
	 */
	auto targetPart = Storage::findPartition(F("spiffs0"));
	if(!targetPart) {
		// Not found: presumably already upgraded
		return MigrateResult::alreadyUpgraded;
	}
	if(targetPart.address() != 0x100000) {
		// Bad offset, unsafe to proceed
		return MigrateResult::failure;
	}

	/*
	 * Locate the source SPIFFS partition and mount it
	 */
	auto sourcePart = Storage::findPartition(F("spiffs1"));
	if(!sourcePart) {
		// Cannot proceed
		return MigrateResult::failure;
	}

	std::unique_ptr<IFS::FileSystem> srcFilesystem{IFS::createSpiffsFilesystem(sourcePart)};
	if(!srcFilesystem) {
		return MigrateResult::failure;
	}
	if(srcFilesystem->mount() < 0) {
		return MigrateResult::failure;
	}

	/*
	 * Construct new partition information so we can mount it as LittleFS and extend the size
	 * This does not change the on-flash structures; we update those later when everything's
	 * been successfully copied over.
	 */
	using Partition = Storage::Partition;
	Partition::Info info(F("lfs0"), Partition::SubType::Data::littlefs, targetPart.address(), 0x200000);
	targetPart = Partition(*Storage::spiFlash, info);

	/*
	 * Create the target LittleFS filesystem, format then mount
	 */
	std::unique_ptr<IFS::FileSystem> dstFilesystem{IFS::createLfsFilesystem(targetPart)};
	if(!dstFilesystem) {
		// Cannot proceed
		return MigrateResult::failure;
	}
	if(dstFilesystem->format() < 0) {
		return MigrateResult::failure;
	}
	if(dstFilesystem->mount() < 0) {
		return MigrateResult::failure;
	}

	// Copy files
	IFS::FileCopier copier(*srcFilesystem, *dstFilesystem);

	copier.onError([&](const IFS::FileCopier::ErrorInfo& info) -> bool {
		IFS::FileSystem::Info fsi{};
		info.fileSys.getinfo(fsi);

		/*
		 * SPIFFS doesn't have a root directory entry
		 */
		if(fsi.type == IFS::FileSystem::Type::SPIFFS && info.operation == IFS::FileCopier::Operation::open &&
		   !info.path) {
			return true;
		}

		Serial << fsi.partition.name() << ": " << info << endl;
		return false;
	});

	if(!copier.copyDir(nullptr, nullptr)) {
		// Copy failed
		return MigrateResult::failure;
	}

	//
	// TODO: Consider verifying file contents
	//

	//
	// TODO: Write modified partition table to flash
	//

	// Reboot
	System.restart();
	return MigrateResult::success;
}

[pljakobs]

oh wow, I didn't expect you to actually write some code here :o
I have, over the weekend, succeeded to change the partition table and write it back to flash, albeit in a not very portable way, so very much limited to my project.
Your code is at least 10 times more elegant, so I will certainly use parts of it.
However, this morning, while going through the IFS documentation again, I noticed a paragraph that says LittleFS is predominantly for devices with few write operations and SPIFFS has actually better wear levelling?
One of the reasons to change the file system was the fact that I've seen many flash failures over the years. The firmware saves the resulting {h,s,v} json after every fade of the lights, which, for example if the lights are coupled to a motion sensor, can be dozens a day approaching 10.000 in a year. I have had controllers fail to mount their file system after a year and a bit, and my assumption was that this was due to excessive writes to a small number of flash cells.
I would assume with full static or even just dynamic wear levelling, re-writing this very small file (maybe max 64Bytes, so well shorter than one block) should not be an issue on a partition that is 768kB in size.
The new application will actually store more dynamic content on flash:

• current color: as said above, single file, being written to possibly many dozen times a day
• presets, groups and scenes: dozens of small files, predominantly being written once and read multy
• hosts: dozens of small files: dozens of small files, written once. I would love to be able to add a "last seen" timestamp here, but that might make them written as often as the current color.
• general config: medium sized file (few 1000s of bytes), while there are a number of front end configurations that trigger a write to the config file, I don't expect that to happen more than a few 100 times during the life time of a device.

The only concern therefore is the current color file (which is used to restore the last color after a power outage like an external switch). If I had eeprom, I would certainly write it there and be done with thinking about flash wear.

But back to the initial thought: I would have thought that with a partition of this size and around 300kB unused even with the old webapp in the spiffs, this should not have worn out in just 10.000 cycles.

What am I missing here? I have read through the design doc on LittleFS and think it's well designed to provide good wear resistance while maybe not providing the best wear levelling. My assumption was, that in my case, especially with the partitions being rather large and sparsely used, it would easily be able to outlast any reasonable expected life time for the device (probably 20 years plus, since the initial devices have already been around since 2015 or so) even under the given conditions.

Apart from all this, I was hoping for LFS to help me save some precious RAM, which would be most welcome.

[mikee47]

oh wow, I didn't expect you to actually write some code here :o

Choice of language to sift through a problem: English, or C++ ? LOL.

... noticed a paragraph that says LittleFS is predominantly for devices with few write operations and SPIFFS has actually better wear levelling?

Can you point me to that paragraph? Not sure that's entirely accurate...

... I have had controllers fail to mount their file system after a year and a bit, and my assumption was that this was due to excessive writes to a small number of flash cells.
... I would have thought that with a partition of this size and around 300kB unused even with the old webapp in the spiffs, this should not have worn out in just 10.000 cycles.

So more likely it's for some other reason. For example, power instability (fluctuation / spike / power cut) during write operations can corrupt SPIFFS: A primary design goal for littlefs is resiliency in this situation.

If you want to further assess the suitability of littlefs for your application, take a look at the Inspect sample application included in the LittleFS Component. Specifically, the fstest() function uses the default IFS profiler to track read/write/erase cycles. If you wanted to investigate wear levelling you could write your own profiler to track which flash blocks get erased over time. Running on host you could evaluate many years very quickly.

[pljakobs]

Choice of language to sift through a problem: English, or C++ ? LOL.

fair point :-)

Can you point me to that paragraph? Not sure that's entirely accurate...

I might have misinterpreted that, but to me, the paragraph "Configuration Filesystem" indicates that spiffs may have better wear levelling
https://sming.readthedocs.io/en/latest/_inc/Sming/Components/IFS/index.html

So more likely it's for some other reason. For example, power instability (fluctuation / spike / power cut) during write
operations can corrupt SPIFFS: A primary design goal for littlefs is resiliency in this situation.

I feel that an electrical failure would be highly unlikely since, in my case for example, the devices have been always on. Sure there could be spikes, but realistically, the Esp8266 has an on board 3.3V local switching power supply and off board, there's a 12V wall plug. I've looked at the quality of the 3.3V supply quite carefully (designed the switching circuit myself) and could not find any transients from the primary side to come through.
Also, disabling the color save function solved the problem as far as I can tell. I assume it's more likely that a file system root pointer has gone bad. I saw that LittleFS has a nice solution for chaning the root pointer as rarely as possible with the double root node and the actual root pointer only having to be updated if one of the root node pages has gone bad.

[mikee47]

... the paragraph "Configuration Filesystem" indicates that spiffs may have better wear levelling

It's intended to address the specific issue of storing configuration data but doesn't do it very well :-( I need to tidy up that documentation!

[mikee47]

I've added a feature/wear-level-check branch to the LittleFS library, comparing behaviour of LittleFS vs. SPIFFS.

[pljakobs]

I finally had the time to look at this and compare the behaviour.
I ran it 100.000 times and the results are - interesting.
First: both lfs and spiffs both survived the torture.
Second: spiffs seems to better balance across the full device, but lfs seems to much better avoid writing stuff.
Three blocks were written >73000 times, most others only about 15000 times, about two thirds of blocks were not touched at all. while spiffs seems to write about 4x as often to even than to odd blocks.
All in all, the results indicated that the main advantage of lfs is the copy-on-write architecture, which avoids many writes (at the price of making write access quite a bit slower)

LFS:

Read: 
  0: 16115288 16076592       0       0 1000510    2426       0       0
  8:       0       0       0       0       0       0       0    2426
 16: 7237340       0       0    2422       0       0       0       0
 24:       0       0       0       0       0       0       0       0
 32:       0       0       0    2422       0    2422       0       0
 40:    2422       0    2422       0       0       0       0       0
 48:       0       0       0    2422       0    2422       0  864898
 56:    2426       0       0       0       0    2422       0       0
 64:       0       0  232624    2426       0       0       0    2422
 72:       0   15496   15496   15496   15496   15496   15496   15496
 80:   15496   15496   15496   15496   15496 16191880 32424287   15496
 88:   15496   15496   15496   15496   15496   15496   15496   15496
 96:   15496   15496   15496   15496 32471835   15496 32455176   15496
104: 16239554   15496   15496   16734   15496   15496   15496   15496
112:   15496   15488   15488   15488   15488   15488   15488   15488
120:   15488   15488   15488   15488   15488   11616    7744       0
128:       0       0    2422       0       0       0       0    2422
136:       0       0       0       0       0       0       0       0
144:    2422       0       0       0       0       0       0       0
152:       0       0       0    2422  239902    2426       0       0
Write: 
  0:   31821   31793       0       0       0       0       0       0
  8:       0       0       0       0       0       0       0       0
 16:       0       0       0       0       0       0       0       0
 24:       0       0       0       0       0       0       0       0
 32:       0       0       0       0       0       0       0       0
 40:       0       0       0       0       0       0       0       0
 48:       0       0       0       0       0       0       0       0
 56:       0       0       0       0       0       0       0       0
 64:       0       0       0       0       0       0       0       0
 72:       0   15496   15496   15496   15496   15496   15496   15496
 80:   15496   15496   15496   15496   15496   44368   73240   15496
 88:   15496   15496   15496   15496   15496   15496   15496   15496
 96:   15496   15496   15496   15496   73360   15496   73360   15496
104:   44488   15496   15496   15547   15496   15496   15496   15496
112:   15496   15488   15488   15488   15488   15488   15488   15488
120:   15488   15488   15488   15488   15488   11616    7744       0
128:       0       0       0       0       0       0       0       0
136:       0       0       0       0       0       0       0       0
144:       0       0       0       0       0       0       0       0
152:       0       0       0       0       0       0       0       0
Erase: 
  0:     249     249       0       0       0       0       0       0
  8:       0       0       0       0       0       0       0       0
 16:       0       0       0       0       0       0       0       0
 24:       0       0       0       0       0       0       0       0
 32:       0       0       0       0       0       0       0       0
 40:       0       0       0       0       0       0       0       0
 48:       0       0       0       0       0       0       0       0
 56:       0       0       0       0       0       0       0       0
 64:       0       0       0       0       0       0       0       0
 72:       0    1937    1937    1937    1937    1937    1937    1937
 80:    1937    1937    1937    1937    1937    1796    1655    1937
 88:    1937    1937    1937    1937    1937    1937    1937    1937
 96:    1937    1937    1937    1937    1655    1937    1655    1937
104:    1796    1937    1937    1937    1937    1937    1937    1937
112:    1937    1936    1936    1936    1936    1936    1936    1936
120:    1936    1936    1936    1936    1936    1452     968       0
128:       0       0       0       0       0       0       0       0
136:       0       0       0       0       0       0       0       0
144:       0       0       0       0       0       0       0       0
152:       0       0       0       0       0       0       0       0

SPIFFS:

Read: 
  0:       0       0       0       0       0       0       0       0
  8:       0       0       0       0       0       0       0       0
 16:       0       0       0       0       0       0       0       0
 24:       0       0       0       0       0       0       0       0
 32:       0       0       0       0       0       0       0       0
 40:       0       0       0       0       0       0       0       0
 48:       0       0       0       0       0       0       0       0
 56:       0       0       0       0       0       0       0       0
 64:       0       0       0       0       0       0       0       0
 72:       0       0       0       0       0       0       0       0
 80:       0       0       0       0       0       0       0       0
 88:       0       0       0       0       0       0       0       0
 96:       0       0       0       0       0       0       0       0
104:       0       0       0       0       0       0       0       0
112:       0       0       0       0       0       0       0       0
120:       0       0       0       0       0       0       0       0
128:       0       0       0       0       0       0       0       0
136:       0       0       0       0       0       0       0       0
144:       0       0       0       0       0       0       0       0
152:       0       0       0       0       0       0       0       0
Write: 
  0:   53000   15413   53271   15502   53086   15323   53148   15348
  8:   53423   15350   53556   15382   53488   15258   53750   15220
 16:   54594   15524   54432   15487   54007   15316   54207   15341
 24:   54570   15624   54222   15390   54225   15404   53745   15157
 32:   53952   15165   54186   15368   54173   15307   54260   15402
 40:   54250   15349   54034   15312   54144   15379   54423   15471
 48:   54248   15357   53968   15300   54114   15317   53914   15275
 56:   53690   15180   53750   15189   54020   15266   54246   15409
 64:   53885   15163   54564   15500   54195   15335   54387   15425
 72:   54289   15412   54150   15366   54176   15364   53967   15252
 80:   54471   15573   54539   15468   54214   15308   54430   15443
 88:   54038   15232   54029   15233   54265   15294   54258   15425
 96:   54382   15467   54382   15360   53681   15071   54345   15440
104:   53926   15193   54323   15361   53917   15222   54278   15370
112:   54154   15278   54366   15402   54219   15429   53780   15176
120:   54179   15384   54197   15363   54101   15315   54665   15628
128:   54356   15488   54167   15413   54061   15253   54353   15441
136:   54311   15368   54535   15460   54362   15417   54188   15374
144:   53765   15236   54131   15461   53746   15374   53372   15281
152:   53290   15374   53338   15487   53431   15546   53033   15509
Erase: 
  0:     192     192     192     192     192     192     192     192
  8:     192     192     193     193     192     192     192     192
 16:     192     192     192     192     192     192     192     192
 24:     192     192     192     192     192     192     192     192
 32:     192     192     192     192     192     192     192     192
 40:     192     192     192     192     192     192     192     192
 48:     192     192     192     192     192     192     192     192
 56:     192     192     192     192     193     193     192     192
 64:     192     192     192     192     192     192     192     192
 72:     192     192     192     192     192     192     193     193
 80:     193     193     194     194     193     193     193     193
 88:     193     193     193     193     193     193     193     193
 96:     193     193     192     192     193     193     193     193
104:     193     193     193     193     193     193     193     193
112:     192     192     192     192     192     192     193     193
120:     193     193     192     192     192     192     192     192
128:     192     192     193     193     193     193     193     193
136:     193     193     193     193     192     192     192     192
144:     193     193     192     192     192     192     192     192
152:     193     193     193     193     192     192     192     192

I find it interesting, that, despite having similar write numbers to at least half of the blocks, SPIFFS does less erase operations.

From the overall design, though, I believe that LFS should deal more gracefully with degradation of flash cells.
Thanks for providing this tool!