Implement BufferedBlockDevice #6757

davidsaada · 2018-04-26T16:56:10Z

Description

Block device allowing smaller read and program sizes for the underlying BD, using a cache.

Pull request type

[ ] Fix
[ ] Refactor
[ ] New target
[X] Feature
[ ] Breaking change

geky

Looks good to me 👍

0xc0170 · 2018-04-27T07:27:50Z

features/filesystem/bd/BufferedBlockDevice.h

+
+
+/** Block device for allowing smaller read and program sizes for the underlying BD,
+ *  using a buffer.


Not sure if this should be extended but first question I have is probably too implementation detailed? the buffer is static, heap? I can see its on the heap.

Why only smaller ? how much smaller ? Is there a range or any limit? What if I have BD that might need bigger reads? What to use?

These might be for documentation or if we can make this more explicit here?

Changed this line a bit. It is actually minimal and not smaller - read and program sizes are fixed to 1.
Reasoning behind such a block device adaptor is the following: Some BDs have read/program sizes that are too big. For instance, the SD card BD has the value of 512 bytes for both. This can make life very hard for users of these BDs, as they will need to add logics for caching data even when they need to read/write small amounts of it. This adaptor solves the problem. One use case (that was the trigger here) was running StoageLite over SD card.

deepikabhavnani · 2018-04-27T14:42:24Z

features/filesystem/bd/BufferedBlockDevice.cpp

+int BufferedBlockDevice::flush()
+{
+    if (!_flushed) {
+        int ret = _bd->program((const void *) _cache, _curr_aligned_addr, _bd_program_size);


~~Query: During init we set _curr_aligned_addr = _bd->size();, if flush is called after init, it will program just the last block of the block device or go beyond the size boundary?~~

Nevermind... Didn't see if (!_flushed)

deepikabhavnani

Looks good to me

kjbracey · 2018-04-30T10:06:14Z

features/filesystem/bd/BufferedBlockDevice.cpp

+int BufferedBlockDevice::flush()
+{
+    if (!_flushed) {
+        int ret = _bd->program((const void *) _cache, _curr_aligned_addr, _bd_program_size);


I think the cast is unnecessary - hope it is, or something unpleasant is happening.

Right. Probably was painted red by Eclipse at some point. Fixed.

kjbracey · 2018-04-30T10:06:31Z

features/filesystem/bd/BufferedBlockDevice.cpp

+        }
+        bd_addr_t offs_in_buf = addr - _curr_aligned_addr;
+        bd_size_t chunk = std::min(_bd_program_size - offs_in_buf, size);
+        memcpy(buf, (const void *)(_cache + offs_in_buf), chunk);


Pointless cast

kjbracey · 2018-04-30T10:06:56Z

features/filesystem/bd/BufferedBlockDevice.cpp

+
+    bd_addr_t aligned_addr = align_down(addr, _bd_program_size);
+
+    uint8_t *buf = (uint8_t *) b;


Should be static_cast<> to check const and assigning to const uint8_t *.

Correct. Fixed.

kjbracey · 2018-04-30T10:07:37Z

features/filesystem/bd/BufferedBlockDevice.cpp

+            // this is covered in the previous condition).
+            prog_buf = buf;
+            if (size == _bd_program_size) {
+                memcpy(_cache, (const void *)buf, _bd_program_size);


Unnecessary cast

kjbracey · 2018-04-30T10:10:53Z

features/filesystem/bd/BufferedBlockDevice.cpp

+    _bd_program_size = _bd->get_program_size();
+
+    if (!_cache) {
+        _cache = new uint8_t[_bd_program_size];


If this alloc is being deferred to the error-returning init, does it make sense to use (nothrow) and return an error on memory failure?

Reason for this being allocated in init is the fact that it relies on the underlying BD's get_program_size, whose value may only be valid after init. Code originally included an assertion for allocation failure, but then was told by @geky that it was redundant, as mbed-os automatically asserts on allocation failures. Is this not correct?

That's correct - it just seems slightly odd to throw given that all other errors in the function return an error code.

On the other hand, it's not that you really want dynamic allocation with check - if it weren't for that get_program_size you'd be doing a throwing new in the constructor, right? So maybe it's just that, deferred.

I'm generally in favour of throwing for unexpected errors, rather than returning an error no-one checks. So fine.

I try to distinguish "fatal" errors (like not being able to allocate at init) from "non fatal" ones (errors in read/write for wrong addresses etc.). At least my code (like StorageLite) checks for error returned by the latter. Maybe I should return the assert for allocation failure just for visibility sake.

I think this is fine - I'll leave it up to you.

I think the question is "would an application want to recover from an OOM in the buffered block device?"

kjbracey · 2018-04-30T10:12:55Z

features/filesystem/bd/BufferedBlockDevice.cpp

+
+int BufferedBlockDevice::deinit()
+{
+    return _bd->deinit();


Should this free the cache for symmetry? Seems reasonable if they've explicitly called deinit and allocation's in init, but maybe there are practical reasons not to.

Well, in this specific block device, there's no difference between freeing the cache (or any other resource for that matter) here or in the destructor. However, tried to conform to other block devices, which let the data live through init/deinit calls.

Ah! The heap block device is special in this case. (It's "pretending" the heap is non-volatile storage.) I would suggest freeing in deinit also.

Gotcha. Moved to deinit.

davidsaada · 2018-04-30T13:35:22Z

Pushed changes following review. Feel free to re-review.

kjbracey · 2018-04-30T13:41:09Z

features/filesystem/bd/BufferedBlockDevice.cpp

+
+int BufferedBlockDevice::deinit()
+{
+    if (_cache) {


Don't need the if - always valid to delete or free null pointers.

Thanks. Removed it.

AnotherButler

Nice hyphenation 👍

AnotherButler · 2018-04-30T19:48:52Z

Should we have a user API page for BufferedBlockDevice in the handbook?

0xc0170 · 2018-05-03T12:48:48Z

Should we have a user API page for BufferedBlockDevice in the handbook?

I believe we should, thus this should wait until PR for the docs is up

0xc0170 · 2018-05-03T12:49:05Z

/morph build

mbed-ci · 2018-05-03T13:20:55Z

Build : SUCCESS

Build number : 1905
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6757/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

0xc0170

Waiting for docs update (if any, otherwise I'll update the review state)

davidsaada · 2018-05-03T16:01:01Z

Will open a documentation PR when back from GEC.

mbed-ci · 2018-05-03T19:28:27Z

Exporter Build : SUCCESS

Build number : 1552
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6757/

mbed-ci · 2018-05-04T02:21:54Z

Test : SUCCESS

Build number : 1723
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6757/1723

0xc0170 · 2018-05-07T10:05:31Z

/morph uvisor-test

mbed-ci · 2018-05-14T17:27:43Z

Build : SUCCESS

Build number : 2008
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6757/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-05-15T05:38:21Z

Test : FAILURE

Build number : 1818
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6757/1818

mbed-ci · 2018-05-15T11:36:08Z

Exporter Build : FAILURE

Build number : 1660
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6757/

0xc0170 · 2018-05-15T13:45:51Z

/morph build

mbed-ci · 2018-05-15T14:18:26Z

Build : SUCCESS

Build number : 2023
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6757/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

cmonr · 2018-05-15T15:01:54Z

/morph build

cmonr · 2018-05-15T15:02:46Z

Grrr... I hate that GitHub pages don't auto-refresh...

mbed-ci · 2018-05-15T15:34:32Z

Build : SUCCESS

Build number : 2025
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6757/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

geky · 2018-05-15T23:12:26Z

features/filesystem/bd/BufferedBlockDevice.cpp

+
+static inline uint32_t align_down(bd_size_t val, bd_size_t size)
+{
+    return val / size * size;


In the future, this guy should definitely have parenthesis

geky · 2018-05-15T23:14:15Z

features/filesystem/bd/BufferedBlockDevice.h

+     */
+    virtual bd_size_t size() const;
+
+protected:


Nonblocker, but is there a reason this is protected?

geky

Looks good to me 👍

mbed-ci · 2018-05-15T23:56:47Z

Test : FAILURE

Build number : 1831
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6757/1831

mbed-ci · 2018-05-16T04:06:37Z

Exporter Build : SUCCESS

Build number : 1674
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6757/

cmonr · 2018-05-16T18:15:45Z

/morph test

cmonr · 2018-05-16T21:06:03Z

/morph test

cmonr · 2018-05-16T21:24:38Z

/morph test

mbed-ci · 2018-05-17T04:17:32Z

Test : FAILURE

Build number : 1850
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6757/1850

cmonr · 2018-05-18T03:00:44Z

/morph test

mbed-ci · 2018-05-18T06:56:22Z

Test : SUCCESS

Build number : 1869
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6757/1869

adbridge · 2018-05-21T14:02:45Z

@kjbracey-arm @0xc0170 Are you guys happy now with this ?

geky previously approved these changes Apr 26, 2018

View reviewed changes

geky requested a review from deepikabhavnani April 26, 2018 20:18

0xc0170 added the needs: review label Apr 27, 2018

0xc0170 requested a review from AnotherButler April 27, 2018 07:25

0xc0170 reviewed Apr 27, 2018

View reviewed changes

davidsaada dismissed geky’s stale review via ee7b040 April 27, 2018 07:33

davidsaada force-pushed the david_buffered_bd branch from 6732ca9 to ee7b040 Compare April 27, 2018 07:33

deepikabhavnani reviewed Apr 27, 2018

View reviewed changes

deepikabhavnani previously approved these changes Apr 27, 2018

View reviewed changes

kjbracey reviewed Apr 30, 2018

View reviewed changes

adbridge added needs: work and removed needs: review labels Apr 30, 2018

davidsaada dismissed deepikabhavnani’s stale review via df8e360 April 30, 2018 13:34

davidsaada force-pushed the david_buffered_bd branch from ee7b040 to df8e360 Compare April 30, 2018 13:34

kjbracey previously approved these changes Apr 30, 2018

View reviewed changes

AnotherButler previously approved these changes Apr 30, 2018

View reviewed changes

0xc0170 added needs: CI and removed needs: work labels May 3, 2018

0xc0170 previously requested changes May 3, 2018

View reviewed changes

cmonr added needs: work needs: preceding PR labels May 7, 2018

cmonr added the release-version: 5.9.0-rc1 label May 14, 2018

davidsaada mentioned this pull request May 15, 2018

Add the StorageLite feature #6915

Closed

geky reviewed May 15, 2018

View reviewed changes

geky approved these changes May 15, 2018

View reviewed changes

0xc0170 added needs: review and removed needs: work labels May 20, 2018

kjbracey approved these changes May 21, 2018

View reviewed changes

cmonr added ready for merge and removed needs: review labels May 21, 2018

0xc0170 merged commit fc2f7ff into ARMmbed:master May 22, 2018

0xc0170 removed the ready for merge label May 22, 2018

davidsaada deleted the david_buffered_bd branch July 9, 2018 13:00



		/** Block device for allowing smaller read and program sizes for the underlying BD,
		* using a buffer.


		bd_addr_t aligned_addr = align_down(addr, _bd_program_size);

		uint8_t buf = (uint8_t ) b;

Implement BufferedBlockDevice #6757

Implement BufferedBlockDevice #6757

Uh oh!

Conversation

davidsaada commented Apr 26, 2018 • edited by adbridge Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Uh oh!

geky left a comment

Choose a reason for hiding this comment

Uh oh!

0xc0170 Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepikabhavnani Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepikabhavnani left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidsaada commented Apr 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AnotherButler left a comment

Choose a reason for hiding this comment

Uh oh!

AnotherButler commented Apr 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xc0170 commented May 3, 2018

Uh oh!

0xc0170 commented May 3, 2018

Uh oh!

mbed-ci commented May 3, 2018

Triggering tests

Uh oh!

davidsaada commented Apr 26, 2018 •

edited by adbridge

Loading

0xc0170 Apr 27, 2018 •

edited

Loading

deepikabhavnani Apr 27, 2018 •

edited

Loading

AnotherButler commented Apr 30, 2018 •

edited

Loading