Cloudtrail add actor and target

[romulets]

This is a re-open of #11245, that had to be closed because a full commit history rewrite in the integration main.

Background

Elastic Cloud Security Team has been focusing, this past year, on Cloud Detection and Response (CDR). One of the first steps towards the CDR vision is to enhance investigation workflows for the Cloud Security use-case in SIEM.

As part of enhancing investigation workflows it's necessary to be able to correlate events and entities. Meaning, if an alert is triggered on the ec2 instance i-000000000, it is of great value to easily be able to search all the events related to that entity, across multiple indices, with one query. Therefore we are working on extracting entities and enabling them to be correlated.

What is an entity?

An "entity" in our context refers to any discrete component within an IT environment that can be uniquely identified and monitored. This broad term encompasses both managed and unmanaged elements.

The term "entity" is broader than the current set of available fields under related. Although ip, user and hosts can be identities, there is a lack of space to represent messaging queues, load balancers, storage systems, databases and others. Therefore the proposal to add a new field.

The proposed structure

There are two fields being added on this PR:

• actor.entity.id captures entities that started the event, the actors
• target.entity.id captures entities that were affected by the event. Being that created, updated, listed. We try to do as much as possible with the data present in the event.

Decisions made on the Painless Script

Structure

The painless script turned very large. There are essentially three parts to it:

• Definition of helper functions. They are meant to facilitate the handling of the collections (related, actor and target).
• Definition of enriching functions per AWS service. Even though there is no defined structure to requestParameters and repsonseElements, there is, usually a somewhat coherent structure per AWS service. I believe such separation brings better reading, creates a better headspace once working in a specific service and also breaks down the huge if else chain present in the previous state of the code
• Calling functions and setting fields.

Why TreeSet as datastructure to hold related, actor, target.

There are two properties that this script must have:

• Values must be unique
• Values must be sorted (for testability and consistency on production)

Previously I had ensured both properties on "post processing", at the end of the script. Now it's ensured by the data structure itself.

I have not performance tested myself, but the usage of TreeSet should improve the time complexity of the algorithm, since we sort data on add, and previously we had to sort afterwards. I couldn't find a reliable source for time complexity of TreeSet.add vs Collections.sort - and honestly, the size of the list is so small that might not even matter.

Amount of tests

The testing was essential to me to validate what I was doing, to verify each output. And I would like to keep the tests for future reference and ensuring we are not changing anything by mistake. But the tests are starting to get slow. Specially if you compare with other integrations, such as okta.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

Approving change in packages/aws/docs/cloudtrail.md as codeowner.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[agithomas]

For the pipeline segment, do you still want to keep it? This may be irrelevant following the change mentioned here. Please re-check

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: b84a382

Failed CI Steps

• Check integrations aws

History

• 💔 Build #23748 failed 55b325c
• 💔 Build #23747 failed fc00ced
• 💚 Build #23286 succeeded bf1fd33
• 💚 Build #23110 succeeded 8cd3c03

[agithomas]

LGTM!

[elastic-vault-github-plugin-prod]

Package aws - 2.44.0 containing this change is available at https://epr.elastic.co/package/aws/2.44.0/