Skip to content

Cloudtrail add actor and target #11245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

romulets
Copy link
Member

@romulets romulets commented Sep 25, 2024

Background

Elastic Cloud Security Team has been focusing, this past year, on Cloud Detection and Response (CDR). One of the first steps towards the CDR vision is to enhance investigation workflows for the Cloud Security use-case in SIEM.

As part of enhancing investigation workflows it's necessary to be able to correlate events and entities. Meaning, if an alert is triggered on the ec2 instance i-000000000, it is of great value to easily be able to search all the events related to that entity, across multiple indices, with one query. Therefore we are working on extracting entities and enabling them to be correlated.

What is an entity?

An "entity" in our context refers to any discrete component within an IT environment that can be uniquely identified and monitored. This broad term encompasses both managed and unmanaged elements.

The term "entity" is broader than the current set of available fields under related. Although ip, user and hosts can be identities, there is a lack of space to represent messaging queues, load balancers, storage systems, databases and others. Therefore the proposal to add a new field.

The proposed structure

There are two fields being added on this PR:

  • actor.entity.id captures entities that started the event, the actors
  • target.entity.id captures entities that were affected by the event. Being that created, updated, listed. We try to do as much as possible with the data present in the event.

Decisions made on the Painless Script

Structure

The painless script turned very large. There are essentially three parts to it:

  • Definition of helper functions. They are meant to facilitate the handling of the collections (related, actor and target).
  • Definition of enriching functions per AWS service. Even though there is no defined structure to requestParameters and repsonseElements, there is, usually a somewhat coherent structure per AWS service. I believe such separation brings better reading, creates a better headspace once working in a specific service and also breaks down the huge if else chain present in the previous state of the code
  • Calling functions and setting fields.

Why TreeSet as datastructure to hold related, actor, target.

There are two properties that this script must have:

  • Values must be unique
  • Values must be sorted (for testability and consistency on production)

Previously I had ensured both properties on "post processing", at the end of the script. Now it's ensured by the data structure itself.

I have not performance tested myself, but the usage of TreeSet should improve the time complexity of the algorithm, since we sort data on add, and previously we had to sort afterwards. I couldn't find a reliable source for time complexity of TreeSet.add vs Collections.sort - and honestly, the size of the list is so small that might not even matter.

Amount of tests

The testing was essential to me to validate what I was doing, to verify each output. And I would like to keep the tests for future reference and ensuring we are not changing anything by mistake. But the tests are starting to get slow. Specially if you compare with other integrations, such as okta.

@romulets romulets force-pushed the cloudtrail-add-origin-and-target branch from 241ed25 to b5e160f Compare September 27, 2024 14:15
@romulets romulets force-pushed the cloudtrail-add-origin-and-target branch 3 times, most recently from 6cef63a to 5d506d5 Compare October 10, 2024 11:43
@romulets romulets force-pushed the cloudtrail-add-origin-and-target branch from 883391d to 5c2dce2 Compare October 15, 2024 08:15
@romulets romulets force-pushed the cloudtrail-add-origin-and-target branch 3 times, most recently from d445e1e to 8f088ee Compare October 18, 2024 08:45
@romulets romulets added enhancement New feature or request Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Oct 18, 2024
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Oct 18, 2024

🚀 Benchmarks report

Package aws 👍(11) 💚(6) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
vpcflow 6622.52 5376.34 -1246.18 (-18.82%) 💔
waf 7042.25 4310.34 -2731.91 (-38.79%) 💔

To see the full report comment with /test benchmark fullreport

@romulets romulets force-pushed the cloudtrail-add-origin-and-target branch from 8f088ee to 02e21e3 Compare October 21, 2024 08:16
@terrancedejesus
Copy link
Contributor

@romulets - We will begin reviewing this PR today. Apologies for the latency.

@terrancedejesus
Copy link
Contributor

For GetCallerIdentity, we have found that user.name is fairly consistent and available. However, it can be misleading.

Examples:

user.name: pwncloud-backdoor-user and aws.cloudtrail.user_identity.arn: arn:aws:iam::891377031307:user/pwncloud-backdoor-user
user.name: ec2-instance-role and aws.cloudtrail.user_identity.arn: arn:aws:sts::891377031307:assumed-role/ec2-instance-role/i-0c406f306dc32cfb4

The importance here is the second example where we are getting the identity of the ec2-instance-role, but only as it pertains to i-0c406f306dc32cfb4. What this would indicate to someone is that the source of this API call is from the i-0c406f306dc32cfb4 EC2 instance specifically thus sources from there. This is a unique instance for Assumed Roles specifically.

In the first example, this would indicate that the authenticated user pwncloud-backdoor-user is the source requesting identity information on their behalf regardless of where.

@terrancedejesus
Copy link
Contributor

@romulets - By cross-referencing our attack scenario doc (reference https://github.com/elastic/ia-trade-team/issues/456#issuecomment-2444285976), I noticed the following event.action values or AWS API calls are missing from your prioritized list, simply because they are not used in any existing rules. In our doc we have shared links to existing event docs for these as well as source and target per instructions. Please let us know if there is anything else you need for these.

DescribeSecurityGroups
ListBuckets
DescribeNetworkInterfaces
DescribeRegions
ListRoles
DescribeVpcs
DescribeNetworkAcls
ListUsers
DescribeLoadBalancers
DescribeVolumes
DescribeTrails
ListFunctions
ListTables
ListInstanceAssociations
DescribeDBInstances
GetPolicy
CreatePolicy
ListAttachedRolePolicies
CreateTopic
Subscribe
Converse
CreateDocument
CreateControlChannel
OpenControlChannel
CreateDataChannel
OpenDataChannel
TerminateSession
GetBucketPolicy
ListObjects
HeadObject
GetObject
DeleteObject
DeleteBucket

@romulets
Copy link
Member Author

@terrancedejesus do you have examples of events for those you shared?

@imays11
Copy link
Contributor

imays11 commented Oct 30, 2024

@terrancedejesus do you have examples of events for those you shared?

@romulets examples can be found in the table here. Next to each event.action is a link to a saved search in mostly Tin's cluster. For a few we had to use our TRaDE cluster, the serverless project we gave you access to.

@romulets romulets force-pushed the cloudtrail-add-origin-and-target branch 2 times, most recently from 0aea983 to 3c39730 Compare November 1, 2024 10:04
@romulets romulets marked this pull request as ready for review November 1, 2024 10:07
Copy link

@lalit-satapathy
Copy link
Contributor

Adding @tommyers-elastic, for on any comments on observability use case.

@andrewkroh
Copy link
Member

Is there an shared schema defined for these new root-level namespaces (actor and target)? Typically packages only create documents containing fields from their package name derived namespace (e.g. aws.*) or from ECS. This is the guiding principle that prevents conflicting field definitions in the Fleet package ecosystem.

@romulets romulets force-pushed the cloudtrail-add-origin-and-target branch from 4bb4b5f to 68ba02f Compare December 18, 2024 09:54
@romulets
Copy link
Member Author

/test benchmark fullreport

1 similar comment
@romulets
Copy link
Member Author

romulets commented Jan 8, 2025

/test benchmark fullreport

@romulets romulets requested a review from a team as a code owner January 13, 2025 15:46
@tinnytintin10 tinnytintin10 changed the title Cloudtrail add origin and target Cloudtrail add actor and target Jan 28, 2025
@elasticmachine
Copy link

elasticmachine commented Jan 31, 2025

💔 Build Failed

Failed CI Steps

History

  • 💚 Build #20984 succeeded 275b3152d4a7700fb3248ff462ef1364b3cd97ed
  • 💔 Build #20361 failed 03ff2d8135672100f71ae21932cf29e31a8fa7d4
  • 💔 Build #20358 failed 799fd0b6b53a749a714db8591db89ebdb103568f
  • 💔 Build #20146 failed 1460cab6bdb56d184cb9a411d1d237108e454a87
  • 💔 Build #20144 failed 68ba02fe2b746fa9c35e441de54a2db607b8c375

Copy link

Quality Gate failed Quality Gate failed

Failed conditions
23.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@qcorporation qcorporation requested review from a team as code owners February 4, 2025 03:57
@andrewkroh andrewkroh added Integration:1password 1Password (Partner supported) Integration:abnormal_security Abnormal AI New Integration Issue or pull request for creating a new integration package. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Feb 4, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@qcorporation qcorporation force-pushed the main branch 2 times, most recently from eda4138 to f728ca7 Compare February 5, 2025 22:00
@romulets romulets closed this Feb 10, 2025
@romulets romulets deleted the cloudtrail-add-origin-and-target branch February 10, 2025 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:abnormal_security Abnormal AI Integration:aws AWS Integration:1password 1Password (Partner supported) New Integration Issue or pull request for creating a new integration package. Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants