Fix chunk synchronization #301
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jan 30, 2020
Previously, when an external command used the SCPv2 (chunked=true) protocol via splunklib and returned with 50,000 or more rows for one chunk, it would send two replies intead of one. This caused the external command to get out of sync, sending chunk replies before reading the corresponding request. Fixes splunk#150
amysutedja
force-pushed
the
bugfix/DVPL-6821-chunky
branch
from
ab20cc9 to
dacd7a5
Compare
|
Worked with @aij to dig into this and understand what's going on, and the issue is essentially that the v2 protocol can't deal with partial chunks at all. The fix removes the concept of partial chunking completely for v2. There are a couple of cleanup things we can do, but we'll do them separately. Approving. |
Merged
|
@aij @amysutedja Can this issue cause data from indexers to randomly go missing? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The external search command should be sending one chunk response for every chunk received, but extra flushing in the case of large chunks was producing multiple responses per request.
This changes partial flush() to be a noop for v2, and is more explicit about matching up read_chunk and write_chunk calls.
Fixes #150