TIKA-4595: Add dynamic fetcher/emitter management API to PipesClient #2488
+687
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Added SAVE_FETCHER, DELETE_FETCHER, LIST_FETCHERS, GET_FETCHER commands - Added SAVE_EMITTER, DELETE_EMITTER, LIST_EMITTERS, GET_EMITTER commands - Implemented PipesClient public API methods for runtime configuration - Implemented PipesServer command handlers - Added deleteComponent() and getComponentConfig() to AbstractComponentManager - Added wrapper methods to FetcherManager and EmitterManager - Added remove() method to ConfigStore interface and implementations - All tests passing
- saveFetcher now calls both fetcherManager.saveFetcher() and pipesClient.saveFetcher() - This ensures fetchers are available in the forked PipesServer process - Implemented deleteFetcher to call both managers as well - Fixes FetcherNotFoundException when using dynamic fetchers via gRPC The issue was that fetchers saved via gRPC were only stored in the gRPC server's FetcherManager, but when pipesClient.process() forks a new PipesServer process, that process has its own FetcherManager and doesn't have access to the dynamically created fetchers. Now both are updated.
Contributor
Author
🔧 Critical Fix AddedFixed a bug where dynamically created fetchers via gRPC were not available in the forked PipesServer process. The Problem
The Solution
TestingThe e2e test should now pass - fetchers created via gRPC will be available for document processing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds public API to PipesClient for dynamically managing fetchers and emitters at runtime through PipesServer's ConfigStore.
Changes
Protocol Enhancements
PipesClient API
Added public methods:
saveFetcher(ExtensionConfig)- Create/update fetcherdeleteFetcher(String)- Remove fetcherlistFetchers()- List all fetcher IDsgetFetcherConfig(String)- Get fetcher configurationPipesServer Handlers
Implemented request handlers for all 8 commands with proper error handling and serialization.
Core Infrastructure
deleteComponent()andgetComponentConfig()to AbstractComponentManagerremove()method to ConfigStore interface and implementationsBenefits
Testing
JIRA
https://issues.apache.org/jira/browse/TIKA-4595