Skip to content

aws: optimise cloudtrail ingest pipeline for cases where fruitless work is done #14429

@efd6

Description

@efd6

In #14236 options were added for users to avoid storing data that is not necessary for their use cases. However, during the ingest pipeline's work, these data are still constructed, only to be discarded at the end of the pipeline resulting in unnecessary work and allocations. Now that the options are available, the ingest pipeline should not perform work that it knows will not be stored.

This will require significant delicate refactoring since the ingest pipeline has grown organically with the two major data flow paths being heavily entangled both in the work done and in the subsequent data use.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions