TIKA-4595: Dynamic Fetcher/Emitter Management with ConfigStore Support

[nddipiazza]

Overview

This PR implements dynamic fetcher, emitter, and pipe iterator management for Apache Tika Pipes through a new ConfigStore abstraction, enabling runtime configuration changes without restarts.

Key Features

1. Dynamic Configuration Management API

• New Methods in PipesClient:
  • saveFetcher(FetcherConfig) - Save fetcher at runtime
  • deleteFetcher(String fetcherId) - Remove fetcher
  • updateFetcher(String fetcherId, byte[] config) - Update existing fetcher
  • saveEmitter(EmitterConfig) - Save emitter at runtime
  • deleteEmitter(String emitterId) - Remove emitter
  • updateEmitter(String emitterId, byte[] config) - Update existing emitter
  • savePipeIterator(PipeIteratorConfig) - Save pipe iterator at runtime
  • deletePipeIterator(String iteratorId) - Remove pipe iterator
  • updatePipeIterator(String iteratorId, byte[] config) - Update existing pipe iterator

2. ConfigStore Abstraction

Built-in Implementations:

• InMemoryConfigStore - Fast, ephemeral (default)
• FileBasedConfigStore - JSON persistence, cross-JVM sharing (new)
• IgniteConfigStore - Distributed cache for multi-instance (enhanced)

Storage Support:

• Fetchers
• Emitters
• Pipe Iterators

3. Cross-JVM Configuration Sharing

• PipesClient → gRPC Server → forked PipesServer
• All components saved via gRPC are available to all workers
• Configuration survives process restarts (file/Ignite modes)

4. gRPC API Integration

• SaveFetcher/DeleteFetcher/UpdateFetcher RPC endpoints
• SaveEmitter/DeleteEmitter/UpdateEmitter RPC endpoints
• SavePipeIterator/DeletePipeIterator/UpdatePipeIterator RPC endpoints
• Fully integrated with TikaGrpcServer

Implementation Details

FileBasedConfigStore

• Thread-safe JSON file persistence
• Location: configurable via path parameter
• Atomic writes with temp file + rename
• Automatic initialization from tika-config.json
• Stores all component types in single JSON structure

IgniteConfigStore Enhancements

• Embedded server architecture (no external Ignite needed)
• Client-only mode for workers
• Configurable cache mode (REPLICATED/PARTITIONED)
• JVM arguments for Java module access
• Separate caches per component type

Configuration Examples

File-Based:

{
  "pipes": {
    "configStoreType": "file",
    "configStoreParams": "{\"path\": \"/tmp/tika-config-store.json\"}"
  }
}

Ignite:

{
  "pipes": {
    "configStoreType": "ignite",
    "configStoreParams": "{\"cacheName\": \"tika-config\", \"cacheMode\": \"REPLICATED\"}",
    "forkedJvmArgs": [
      "--add-opens=java.base/java.nio=ALL-UNNAMED",
      "--add-opens=java.base/java.util=ALL-UNNAMED",
      "--add-opens=java.base/java.lang=ALL-UNNAMED",
      "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
      "-DIGNITE_ENABLE_OBJECT_INPUT_FILTER_AUTOCONFIGURATION=false"
    ]
  }
}

Testing

E2E Tests Added

• FileSystemFetcherTest - File-based ConfigStore with dynamic fetcher management ✅
• IgniteConfigStoreTest - Ignite ConfigStore with embedded server ✅
• Document limit feature: -Dcorpa.numdocs=N

Both tests verify:

• Dynamic component creation via gRPC
• Cross-JVM config propagation
• Successful document processing
• Proper cleanup

Backward Compatibility

✅ Fully backward compatible

• Default behavior unchanged (InMemoryConfigStore)
• Existing configs work without modification
• Optional feature - enabled via configStoreType

Related Issues

Fixes: TIKA-4595

Migration Guide

For users wanting dynamic component management:

1. File-based (recommended for single-instance):

   • Add configStoreType: file to pipes config
   • Components persist across restarts

2. Ignite (for multi-instance/distributed):

   • Add configStoreType: ignite
   • Add required JVM arguments
   • Ideal for Kubernetes/multi-pod deployments

Files Changed

• Core: PipesClient, PipesServer, ConfigStore abstraction
• gRPC: TikaGrpcServerImpl with new RPCs for all component types
• Stores: FileBasedConfigStore, IgniteConfigStore enhancements
• Plugins: ExtensionConfig made Serializable
• Managers: FetcherManager, EmitterManager, PipeIteratorManager enhanced
• Tests: E2E tests with document limit feature

Performance Impact

• Minimal overhead (lazy initialization)
• File-based: ~1-2ms per save operation
• Ignite: sub-millisecond after warm-up
• No impact on existing in-memory mode