Skip to content

Checkin worker for extractors #464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Checkin worker for extractors #464

wants to merge 5 commits into from

Conversation

toondaey
Copy link
Contributor

@toondaey toondaey commented Jul 25, 2025

This is a more robust implementation of startup/checkin for extractors. The actual code is just about 359 lines here while the other changes are just tests.
This is supposed to help with:

  • Ensuring that we either don't report any checkin until a start up has been reported or we only report errors not related to tasks run
  • Ensuring that errors and/or task updates are reported chronologically.

It also ensures that a restart of extractors are triggered when the config is changed remotely where the extractor is using a remote config or a fatal error is reported when the extractor encounters an unrecoverable misconfiguration.

The worker also handles chunking of errors/task updates as required by the integrations API.

Copy link

codecov bot commented Jul 28, 2025

Codecov Report

❌ Patch coverage is 77.52809% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.25%. Comparing base (2302bc4) to head (e0cc6c7).

Files with missing lines Patch % Lines
...ite/extractorutils/unstable/core/checkin_worker.py 77.52% 40 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master     #464    +/-   ##
========================================
  Coverage   79.25%   79.25%            
========================================
  Files          42       43     +1     
  Lines        3692     3866   +174     
========================================
+ Hits         2926     3064   +138     
- Misses        766      802    +36     
Files with missing lines Coverage Δ
...ite/extractorutils/unstable/core/checkin_worker.py 77.52% <77.52%> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@toondaey toondaey marked this pull request as ready for review July 28, 2025 06:49
@toondaey toondaey requested a review from a team as a code owner July 28, 2025 06:49
Copy link
Contributor

@einarmo einarmo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good. I think some explanatory comments in the complex parts may be good, and I have some concerns around the locks, but it's taking shape. You will probably need a size exemption.

@toondaey toondaey requested a review from einarmo July 28, 2025 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants