v1.2.0-experimental-sql-indexing-v3

[gtsteffaniak]

Warning

For testing only! Do not use this without knowing the risks.

Main differences from v2 experimental release:

1. batch size reduced from 5000 to 1000 to help keep per transaction memory smaller
2. cache size allocation changes -- minimal difference
3. shrink_memory usage changes
4. fixes to folder size and index status updates.

In the stable/beta releases of FileBrowser Quantum, all memory (RAM) is held in application space. A difference you may notice here is that sqlite can sometimes use more memory on docker stats or total os memory statistics like task manager. But here's an explanation for whats going on and what to look for:

Previously, all memory used by the index was critical and held by the application in a way that would OOM if there was not enough RAM available. With sqlite, its possible theres paged memory which can increase but should not cause OOM because its reclaimable by the OS.

Now, its safe and useful to place limits in docker memory if needed -- and this will actually keep memory lower in docker stats because the os will reclaim the paged cache

---

This discussion was created from the release v1.2.0-experimental-sql-indexing-v3.

[gtsteffaniak]

heres my graph, I am investigating the spike in memory that happens on idle here:

I believe its from db mainatenence purges.

[Kurami32]

Setting the scope to /, restart the container, and configure the scope again makes it work again. But at soon you start doing operations with big folders or files the same happens again.

Here are the logs

[gtsteffaniak]

Thats a great thing to test. The indexing has nothing to do with the scopes, it should work and scopes don't get looked at at all. But scopes come into play when you try to access info from the index -- it sounds like a simple api change for users with scopes are not being honored. I will double check that.

Folder sizes should also get updated on user actions, but sounds like its that scope issue where api actions don't properly apply the scopes for the full index path. I will take a closer look at that.

[Kurami32]

I was testing a bit more, is a bit odd... Is not related with the scopes at all (like you said)

It happens when a new folder is created inside of one of the parent folders in the root of the source, like moving or copying a folder, those actions create a new folder -- The logs start saying that is skipping index updates (like above).

And that made me think that is with the scopes, but is because with scopes you're always inside a folder, not in the real root (/) of the source.

I'll record a video in a while, but first I test more to be completely sure. Since I'm seeing that is skipping updates for / too but only in one of the sources.

[Kurami32]

2025-12-22.14-32-21.mp4

2025-12-22.14-56-39.mp4

The folder size became negative for a moment too

2025-12-22.14-32-21.online-video-cutter.com.mp4

[gtsteffaniak]

Ahhh during multiple folders copied ? I didn't test that yet. There is a brand new recursive folder size update function AND new functionality for refresh index info on copy move.

It's probably an issue with how they interact. Thanks for the videos I will use that to replicate and fix the behavior.

[Daveedecoder]

What's this all about

Does it relate to laravel project ???

[Kurami32]

Hi, and nope, is not related with Laravel or PHP.

This is a discussion about an experimental build/release of Filebrowser Quantum :)

[gtsteffaniak]

I think I am mostly satisfied with the memory usage at this point, now just to squash the few bugs related to index paths and add a few playwright tests to ensure it always works.

So I tested a few different scenarios:

medium: 2 sources configured, both SSD based and roughly 1 minute to index

large: 3 sources configured, including 1 HDD with lots of files. This large configuration showcases what adding limits to the docker deploy configuration could do. Since the memory usage is now primarily paged memory , its safe and helpful to use limits. I applied a 100 MB limit and 250MB limit to show the cases. I believe I will recommend a 500 MB limit in the end as a general rule and lower limit only for resource constrained environments.

And finally, a test showing 1 small source, very little difference:

[gtsteffaniak]

Looks like I see the container shutdown on the 100mb limit option when navigating some folders. Strange it's a graceful shutdown not an OOM kill like I would expect if something was done to hit the limit.

100mb is obviously too small for a hard limit. But I would like to be able to give a minimum ram requirement that users could put limits on.

[Kurami32]

btw @gtsteffaniak the ram is still increasing when doing uploads, and also when downloading large folders with lots of files inside. The more uploads I do, the ram increases more. Is normal?

NAME              CPU %     MEM USAGE / LIMIT     NET I/O           BLOCK I/O
fbquantum-sql     0.00%     850.7MiB / 7.553GiB   7.21GB / 11.7GB   10.1GB / 8.08GB

After a while decreases a bit (making it look like spikes in the graph), but the ram is still high and "stacks" there. Basically each spike was a upload except for the last one that was a folder download. I'm not sure how much more it can increase.

If matters, I have maxConcurrentUploads: 10 and UploadChunkSizeMb: 100 in upload settings.

---

But I think what is increasing the cache size - The RSS of the filebrowser process is ~277Mb. Tho I'm not sure how to do calculations to know the real ram usage.

~ $ cat /proc/1/status | grep -E "VmSize|VmRSS|VmSwap"
VmSize:	2007244 kB  # Virtual Memory (~2GB).
VmRSS:	 277880 kB  # RSS (~277Mb).
VmSwap:	      0 kB