-
Notifications
You must be signed in to change notification settings - Fork 5
Checkin worker for extractors #464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #464 +/- ##
========================================
Coverage 79.25% 79.25%
========================================
Files 42 43 +1
Lines 3692 3866 +174
========================================
+ Hits 2926 3064 +138
- Misses 766 802 +36
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks pretty good. I think some explanatory comments in the complex parts may be good, and I have some concerns around the locks, but it's taking shape. You will probably need a size exemption.
This is a more robust implementation of startup/checkin for extractors. The actual code is just about 359 lines here while the other changes are just tests.
This is supposed to help with:
It also ensures that a restart of extractors are triggered when the config is changed remotely where the extractor is using a remote config or a fatal error is reported when the extractor encounters an unrecoverable misconfiguration.
The worker also handles chunking of errors/task updates as required by the integrations API.