[Cloudtrail] flattened fields response_elements, response_elements, additional_eventdata increase storage

[norrietaylor]

After investigating storage costs for the cloud trail integration, it was shown that we are storing some large fields multiple times.

cloudtrail.response_elements, cloudtrail.request_elements , cloudtrail.additional_eventdata fields.

We store them five times: _source, keyword, text, and then flattened, which stores them twice (keyed and unkeyed)

After discussion with @efd6, the recommendation is to add a configuration value to disable the flattened fields.

[strawgate]

I generally think these are useful because of the flattened fields -- I think the text field is somewhat questionable but somewhat makes sense, I'm struggling to understand however what someone does with 4kb of json in a keyword field

[leandrojmp]

A feedback as user, I find them pretty useful and we use both the flattened and text field in security rules for different cases and needs.

If I understand correctly the proposal is to add a way for the user to disable it if wanted, similar to what is done if the user wants to keep the original event or duplicated fields in some integrations, right?

[efd6]

I'd like to hear feedback. So far, what I have is an option to not retain the flattened field group. This is slightly more involved than you might expect, since it's not just deleting it at the end of the pipeline, so that we don't do fruitless work. If we need to have multiple of these, I'm less inclined to have multiple options, preferring to only have the one, since it avoids ingest work that's not otherwise needed, but give advice in the package documentation to add an @custom pipeline that deletes fields that are not wanted.

[strawgate]

A feedback as user, I find them pretty useful and we use both the flattened and text field in security rules for different cases and needs.

But are you using the keyword field? How are you using the keyword field for visualizations or aggregations if its 4-8kb of stringified json?

[leandrojmp]

@efd6 @strawgate what exactly is being considered for removal/changes?

The 2 main fields from Cloudtrail are request_parameters and response_elements, those are present in two ways, one is a flattened form with the entire json like aws.cloudtrail.flattened.request_parameters and the other is in a multi-field as aws.cloudtrail.request_parameters for example.

Both are required, the aws.cloudtrail.request_parameters we use more the .text field to search for strings inside of it, the keyword field is used sometimes when the value is just a single keyword in some searchs/dashboards.

In our use case both formats, flattened and text, are required, the aws.cloudtrail.request_parameters could be changed to be just a text/match_only_text, but if it is entirely removed it would break a lot of things.

Also, both aws.cloudtrail.request_parameters and aws.cloudtrail.flattened.request_parameters are currently used in Built-in Elastic detection rules.

The aws.cloudtrail.request_parameters is used in some ESQL hunting rules and EQL security rules if I'm not wrong, so the text version for this field needs to exist, and the aws.cloudtrail.flattened.request_parameters is used in other types of security rules as well.

[strawgate]

Do you by chance have an example of it containing a single value that is then usable as a keyword?

Right now the keyword stores up to 8kb of json.

Keyword fields are stored deduplicated but uncompressed in the inverted index, and so large high cardinality keywords (which 8kb of json is) are expensive to store and, in my opinion, of very suspect utility.

If I could wave a magic wand:

1. Shorten keyword ignore-above to 256
2. Make the keyword/text and flattened fields both optionally disabled

[leandrojmp]

Being honest I'm failing to get what is the issue here.

I do not use the aws.cloudtrail.request_parameters or aws.cloudtrail.response_elements as keyword, I use the multi-field .text, like aws.cloudtrail.request_parameters.text and aws.cloudtrail.response_elements.text and the aws.cloudtrail.flattened.* fields.

The keyword mapping for this field can ben indeed ignored because of the size, but the individual leafs in the flattened version still work, and also the .text field also works.

Example:

So, aws.cloudtrail.request_parameters and aws.cloudtrail.response_elements as keyword may be removed, I think, but it needs to exist as a text or match_only_text and the aws.cloudtrail.flattened.* version of both fields also needs to exist.

Removing the aws.cloudtrail.flattened.* would probably break a lot of AWS Built-in Security rules.

EDIT:

Just some numbers, that I've got, on the last 7 days I've got something close to 1 billion Cloudtrail events and the distribution of events with fields marked as ignored are:

• Around 100 millions for the field aws.cloudtrail.response_elements
• Around 2 millions for the field aws.cloudtrail.request_parameters
• Less than 200 fo the field aws.cloudtrail.additional_eventdata.
• No events had any aws.cloudtrail.flattened.* leaf as ignored.

And per my last comment, I don't think that they should be removed anymore, it would break a lot of things.

[norrietaylor]

It sounds to me like there is broad agreement with the following:

So, aws.cloudtrail.request_parameters and aws.cloudtrail.response_elements as keyword may be removed, I think, but it needs to exist as a text or match_only_text and the aws.cloudtrail.flattened.* version of both fields also needs to exist.

Removing the aws.cloudtrail.flattened.* would probably break a lot of AWS Built-in Security rules.

If this is the case, I will change the description above to reflect this new scope of work.

[leandrojmp]

Just some updates since I was curious and made some tests.

The field request_parameters from the Cloudtrail events dataset, and the same thing applies to the other two similar fields, is indexed multiple times as the following:

• aws.cloudtrail.flattened.request_parameters: flattened
• aws.cloudtrail.request_parameters: keyword (with the max value for ignore above)
• aws.cloudtrail.request_parameters.text: text (is a multi-field from the keyword field)

Both aws.cloudtrail.flattened.request_parameters leaf fields and aws.cloudtrail.request_parameters are used in different types of built-in security rules using different query languages.

If for example the aws.cloudtrail.request_parameters is changed from a mult-field into a text only field this would cause some issues and break stuff.

It would not generated a conflict in a way that Kibana normally complains, but the main things that would happen are:

• ESQL rules using aws.cloudtrail.request_parameters would stop working until there are no more indices with different mappings, they can work on keyword or on text, but they do not work if different backing indices have different mappings, at least not the examples I tested.
• EQL rules would not work anymore, some of them use functions that only work on keyword fields.

If wanted I can provide the steps to replicate those issues.

Considering that Cloudtrail may be a critical dataset for some users and need to be kept for some time in the cluster, this can have a huge impact.

As a user I do not see any benefit on changing any of the mappings for those fields.

Also, I'm speaking with a Security use case in mind, Cloudtrail is heavily used in Security use cases.

[efd6]

So that I can understand the decision that was made here, ISTM that there is nothing to do since any changes will break things. I can add the options to remove things on the optimistic assumption that users will not break themselves, but I think that is likely not a good idea.

[leandrojmp]

@efd6 I don't think that anything should be changed on the mappings for those fields, they are mapped and stored multiple times because they are used in different ways.

The user already has options to remove the wanted fields if they want, changing the integration to expose a new option to allow the user to disable the flattened fields or maybe the multi-fields would need to inform that some Security rules could stop working, which could create some confusion.

[efd6]

@leandrojmp Yeah. ISTM that the answer here is to do nothing. There is an option already to achieve what an option would, that is to add the relevant remove processors to an @custom pipeline. In the case that someone is doing that they are presumably aware of what they are doing.