Filesystem bugs, including volume corruption

[bmcdonnell-ionx]

Description

• Type: Bugs
• Priority: Major?

---

Bug

Target
LPC4088

Toolchain:
GCC_ARM - yes
ARM|IAR - untested

Toolchain version:

mbed-cli version:
1.2.2

mbed-os sha:
2b4ff78

Expected behavior

1. readdir() returns NULL to terminate a directory listing.
2. FATFileSystem writes files to SD Card at a relatively consistent speed.
3. FATFileSystem never corrupts the file system on the SD Card.

Actual behavior

1. When operating on a directory on a FAT32 volume containing the maximum number of files per directory (65534), readdir() never returns NULL.
2. Increasing the number of files in a directory slows down FATFileSystem writes.
3. Under certain cirumstances, FATFileSystem (or SDBlockDevice?) corrupts the file system on the SD Card. (i.e. It can no longer be read by FATFileSystem, nor by a Windows PC.)

Steps to reproduce

I forked ARMmbed/mbed-os-example-filesystem to create demo code for items 1 and 2. There are tags (linked below) for each of these. I don't have code to share for 3 yet, but I give details below.

All testing was done with an Embedded Artists' LPC4088 QuickStart Board connected to an LPC4088 Experiment Base Board (EBB). The EBB is used for its SD Card slot. Jumpers are configured to use the SPI interface to the SD Card (per Figure 7 in the EBB user guide (PDF)).

I made a small patch to mbed OS so that GCC_ARM will malloc() from the LPC4088's external SDRAM like the mbed online compiler does. Demo programs 1 & 2 use megabytes of memory for a HeapBlockDevice. You should be able to replace this with an SDBlockDevice to avoid the large memory use and test on another target device, but of course this is slower to run.

Of course, add your choice of Serial device to output printf() text.

1. bug01-mbed-os-5.7.1-65534-files-dirent-ne-null (1fa0b0b)
2. bug02a-mbed-os.5.7.1-write-slowdown-empty-files (6ab378b)
   bug02b-mbed-os.5.7.1-write-slowdown-small-files (d891ad3)
3. Using an 8 GB SD Card, write many files to a directory. After each file write, append text via fprintf() to a metadata file in the directory, and a log file in the parent directory.
   • Circumstance 1: Files are exactly 512 KiB each, written via fwrite(). Filesystem is corrupted after writing to the 8160th file. I observed this particular failure mode on at least two units.
   • Circumstance 2: Files are approximately 7.5 MB each, written via fprintf(). Filesystem is corrupted after filling up the volume. Observed on one unit so far.

[0xc0170]

@deepikabhavnani @geky Please review

[bmcdonnell-ionx]

Note, #3 takes about two days to run one scenario on the dev board. I haven't yet tried pre-filling the SD Card, though - probably should. 😄

[bmcdonnell-ionx]

#3 may have something to do with me having a logfile open, the volume fills up from another file, then I try to write to the logfile again. Further investigation required.

[deepikabhavnani]

@bmcdonnell-ionx - Thanks for reporting the issues.
I have looked into 1. and it seems logic at https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/fat/FATFileSystem.cpp#L638 is incorrect

ChanFS reports FR_OK for end of directory, hence the check should be just if condition and not else if

-- else if (finfo.fname[0] == 0) {
++ if (finfo.fname[0] == 0) {
            break;
        } 

I do not have target device with that big HEAP memory, and testing with SD card will take long. Will confirm once I have verified.

[deepikabhavnani]

2. Increasing the number of files in a directory slows down FATFileSystem writes.

Yes it will, as per your example files are created in same directory, and while open call all files are scanned in the directory before creation. See https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/fat/ChaN/ff.cpp#L3046

You can split files in multiple directories to get better performance.

We would appreciate a PR to improve performance here, else we might pick it up in future based on priorities.

[bmcdonnell-ionx]

OK, so @deepikabhavnani is working on #1, and has explained #2.

We would appreciate a PR to improve performance here [#2], else we might pick it up in future based on priorities.

I don't anticipate doing so myself, but for reference, how could one improve the performance here?

[deepikabhavnani]

@bmcdonnell-ionx - @geky might help you with that, else you can add query in ChanFs community (http://elm-chan.org/fsw/ff/bd/) and see if you can get some help.

[bmcdonnell-ionx]

I've published test cases for the following. mbed team (@deepikabhavnani, @geky): Is this enough information for you to troubleshoot the issue?

3. Under certain cirumstances, FATFileSystem (or SDBlockDevice?) corrupts the file system on the SD Card. (i.e. It can no longer be read by FATFileSystem, nor by a Windows PC.)

3. Using an 8 GB SD Card, write many files to a directory. After each file write, append text via fprintf() to a metadata file in the directory, and a log file in the parent directory.
   • Circumstance 1: Files are exactly 512 KiB each, written via fwrite(). Filesystem is corrupted after writing to the 8160th file. I observed this particular failure mode on at least two units.

bug03a-mbed-os-5.7.2-filesystem-corruption (650c265)
bug03b-mbed-os-5.7.2-filesystem-corruption (167f1e1)

Notes:

• I'm using malloc() to allocate a 512 KiB block of memory. With my mbed-os patch, malloc() goes to external memory with GCC_ARM.
• This is imitating an unpublished program of mine where the failure occurred, which is why version a includes faux log and checksum files.
  • Results of some test runs are summarized in the table below. I used identical board pairs with 8 GB SD cards. (Each board pair is denoted as a numbered "unit" in the table.)
• As before, add your Serial device for console output.

Run#	program	Date	Unit#	mbed OS ver#	Result
1	(unpublished)	12/22	2	5.7.0 (maybe 5.7.1)	Failed to re-open the log after writing the file 0x00001fe0.bin (8160 decimal). Writing that file may have failed too. There was also a checksum error when reading back the file two loop iterations before the failure. Filesystem corrupted.
2	(unpublished)	12/22	3	5.7.0 (maybe 5.7.1)	Same as run 1.
3	(unpublished)	1/6	1	5.7.2	Same as run 1, except it failed one iteration earlier (00001fdf.bin).
4	(unpublished)	1/6	2	5.7.2	Same as run 1, except it failed much earlier (0000191f.bin).
5	bug03a (650c265)	1/6	2	5.7.2	Same as run 1.
6	bug03b (167f1e1)	1/7	2	5.7.2	Failed to create 00001fe2.bin. Filesystem corrupted.
7	bug03b (167f1e1)	1/7	3	5.7.2	Same as run 6.

I have another test run executing now without the faux log and checksum files; I'll report results when I have them. If I have time I may try smaller fwrite()s to reduce RAM usage, and see if the results are the same.

[bmcdonnell-ionx]

Test runs of my unpublished program did not fail or corrupt the filesystem on 2 GB or 4 GB SD Cards, which can't fit the number of files it took to get to the failure demonstrated above.

[bmcdonnell-ionx]

I edited my comment above, including the table. I added the version (03b) without the faux log and checksum files.

I'm experimenting with pre-filling the test directory. I'll update later.

[bmcdonnell-ionx]

OK. Finally I published:

• bug03d-mbed-os-5.7.2-filesystem-corruption-prepopulate-lowmem (546b61d) - User pre-fills the SD Card with files for a much quicker test. No external memory required. Add your Serial device for console output. [EDIT: Get rev e below, instead.]

mbed team (@deepikabhavnani, @geky) - is this test enough good enough for you to use to troubleshoot?

This version doesn't require the external memory. And it runs much more quickly - around 15 minutes on my board. But you must pre-populate the files.

To pre-populate the SD Card, with your PC, create a folder fs-test. Inside that folder, create 8128 files of 512 KiB each. To follow my naming convention, they are named with hex numbers, 00000000.bin through 00001fbf.bin. The data doesn't matter; I just used /dev/zero as the source.

A bash script like this should do the trick:

#!/usr/bin/bash

mkdir fs-test
cd fs-test

# create the first file
dd if=/dev/zero of=00000000.bin bs=512K count=1

# make copies
for f in `printf '%08x.bin\n' $( seq 1 8127 )`
do
   echo $f
   cp 00000000.bin $f
done

I recommend you create the folder on your hard drive, so you can just copy it repeatedly for multiple test runs. If using Windows, I recommend using Windows Explorer to copy/paste the folder to the SD Card. (I had an issue where the mbed device isn't seeing all the files under certain circumstances, such as when things were moved from one location to another within the SD Card, or when files were copied to the SD Card using Cygwin. If I can nail it down, I'll report here or in another issue. But Ctrl+C/Ctrl+V on the fs-test folder in Windows Explorer to copy it from the hard drive to the SD Card is working reliably for me on Windows 10.)

I did three test runs, on three different units, and on all three it failed to create file 00001fe2.bin, and then corrupted the filesystem. (i.e. You then put it in your PC, and it tells you to format it.)

(Sorry it took so long. I had tried changing several things at once, and I couldn't reproduce the errors, so I made one change at a time. With test runs taking 1+ days, it took a while...)

[deepikabhavnani]

When operating on a directory on a FAT32 volume containing the maximum number of files per directory (65534), readdir() never returns NULL.

@bmcdonnell-ionx Were you able to create 65534 files successfully on HEAP partition/SD card? I was trying with 8GB SD card formatted using FatFilesystem::format() API and using host machine for file creation, below is my observation.

1. Linux system created 32765 files in any directory and then failed with error (No space left on device). Creation of files in another directory was allowed. Read was successful with application on mbed board.
2. Windows created 37,767 files and then failed with error. readdir does not return NULL. Using the same SD card on Linux machine, ls -l hangs and does not list the filenames.

[bmcdonnell-ionx]

Oops - somehow the call to sdram_init() remained in my bug03d release. Not sure how that happened. Skip that one; use this one:

• bug03e-mbed-os-5.7.2-filesystem-corruption-prepopulate-lowmem (1545ebe)

[deepikabhavnani]

@bmcdonnell-ionx - I already did those changes and tried once. Below is the observation, I will be trying again with format using FatFilesystem::format()

Steps:

1. Format on windows
2. 8,128 - 512KB each created on linux
3. Insert in windows and check - Windows complains after linux dump of files as well. Did scan and repair - no error found. (Windows pop error might be because of *.bin file name. TODO: Rename all files to .txt and try later)
4. Edited test to create files from 0x1fc0 (8128) - https://gist.github.com/deepikabhavnani/cdfc66003b5f213b31551bd02338b378

Output:
Open logfile /fs/fs-test/log.txt.
Create test directory /fs/fs-test/00000000.
Create checksums file /fs/fs-test/00000000/checks.txt.
/fs/fs-test/00001fc0.bin 13053 ms - Because overwrites
/fs/fs-test/00001fc1.bin 12666 ms
|
|
/fs/fs-test/00002200.bin 12666 ms and still going on (Killed)

Next Step: Try format with FatFilesystem::format() and repeat steps