Apply a high watermark for pending pings to process from disk #3206
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
badboy
force-pushed
the
push-uypxwvqntuol
branch
from
6554039 to
b9896ae
Compare
travis79
reviewed
Jul 30, 2025
glean-core/src/upload/directory.rs
Outdated
| @@ -267,18 +288,20 @@ impl PingDirectoryManager { | |||
| /// # Returns | |||
| /// | |||
| /// A vector of tuples with the file size and payload of each ping file in the directory. | |||
There was a problem hiding this comment.
Mind updating the doc-comment for the return type to show it now also returns the discarded_files?
glean-core/src/upload/directory.rs
Outdated
| @@ -357,7 +392,7 @@ mod test { | |||
| let directory_manager = PingDirectoryManager::new(dir.path()); | |||
|
|
|||
| // Verify that processing the directory didn't panic | |||
| let data = directory_manager.process_dirs(); | |||
| let data = directory_manager.process_dirs().1; | |||
There was a problem hiding this comment.
nit: I hate how un-descriptive index based access is (.1). Not saying we need to change it but it requires me to go and look up the process_dirs() to understand what .1 vs. .0 means here
There was a problem hiding this comment.
me too. I don't like how we need to thread through the value here now. If we do want this then I might re-do that part.
There was a problem hiding this comment.
I'm all for adding this as a guardrail to keep /pending-pings in check, for what it's worth.
badboy
force-pushed
the
push-uypxwvqntuol
branch
from
b9896ae to
298f3a0
Compare
If we detect more files in the pending pings directory they are removed without processing. Usually we read all files into memory to sort them and then discard the oldest ones if necessary. However if a user has too many files that's very costly (in both processing time and memory usage), so we add a high watermark. This means there's now a case where we can drop arbitrary data. However if a user has _that many pending pings_ some things might have gone awry and the data is less important than keeping the user's application running smoothly.
badboy
force-pushed
the
push-uypxwvqntuol
branch
from
298f3a0 to
7adc7ed
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If we detect more files in the pending pings directory they are removed without processing. Usually we read all files into memory to sort them and then discard the oldest ones if necessary. However if a user has too many files that's very costly (in both processing time and memory usage), so we add a high watermark. This means there's now a case where we can drop arbitrary data. However if a user has that many pending pings some things might have gone awry and the data is less important than keeping the user's application running smoothly.