[Cisco Nexus] Improve timezone and timestamp handling #14504

P1llus · 2025-07-11T10:55:18Z

Proposed commit message

The PR introduces tz_map as an option in the configuration of the integration, similar to tz_map this is specifically for usecases where the vendor provides a timezone format in their logs which is not supported by Java, and is the only way for a user to map these unsupported formats to a proper IANA format on ingest time.

While implementing this there was also several unecessary additions between syslog and vendor timestamps, so I changed that up a bit and updated the system and pipeline tests. Older documents now get missing ECS fields and the dates are parsed correctly also on older logs (which is why older pipeline test output have changes in them).

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

…g even more formats on other date processors

elasticmachine · 2025-07-11T12:40:12Z

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

elastic-vault-github-plugin-prod · 2025-07-11T14:42:10Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elastic-sonarqube · 2025-07-11T14:42:16Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-07-11T14:42:16Z

💚 Build Succeeded

Buildkite Build
Commit: 1d15d60

History

💔 Build #28452 failed 3712469
💔 Build #28450 failed e98d50b
💔 Build #28439 failed d37ee7e

dwhyrock

This looks pretty good, but I had a couple questions.

@taylor-swanson can you take a look at this as well, please?

dwhyrock · 2025-07-11T16:32:51Z

packages/cisco_nexus/changelog.yml

+  changes:
+    - description: Fix bug that did not recognize timestamps to use tz_map override.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14458


PR link needs to be updated

dwhyrock · 2025-07-11T16:39:18Z

packages/cisco_nexus/data_stream/log/elasticsearch/ingest_pipeline/default.yml

@@ -3,126 +3,101 @@ description: Pipeline for processing Cisco Nexus logs.
 processors:
  - set:
      field: ecs.version
-      value: '8.17.0'
-      tag: 'set_ecs_version'
+      value: "8.17.0"


Curious why you're updating the single quotes to double quotes? I do prefer the doubles, but curious if there's some yaml-specific reason to do this.

dwhyrock · 2025-07-11T16:41:18Z

packages/cisco_nexus/data_stream/log/sample_event.json

    },
    "cisco_nexus": {
        "log": {
-            "description": "EARL 0  NF ASIC: Uncorrectable Parity error in Netflow Table.",
-            "facility": "EARL",


Is there a reason why all of these fields are being deleted? Is this part of the processing changes in default.yml, or something else?

taylor-swanson · 2025-07-11T13:01:06Z

packages/cisco_nexus/data_stream/log/manifest.yml

+        required: false
+        show_user: false
+        description: >-
+          A collectiom of timezones found in Cisco Nexus logs (as defined in each `tz_short`), and the replacement value (as defined in each `tz_long`) which should be the full proper IANA Timezone format. This is used to override vendor provided timezone formats that is not supported by Elasticsearch [Date Processors](https://www.elastic.co/docs/reference/enrich-processor/date-processor#date-processor-timezones)


Could we also include offsets as well?

This is a snippet of the description from cisco_asa:

proper IANA time zone or offset (for example, Australia/Sydney or +10:00).

taylor-swanson · 2025-07-16T13:54:11Z

packages/cisco_nexus/data_stream/log/_dev/test/pipeline/test-nexus.log-expected.json

+            "@timestamp": "2025-05-03T13:20:48.739+10:00",
            "cisco_nexus": {
                "log": {
                    "description": "last message repeated 5 time",
                    "ip_address": "10.100.0.34",
                    "priority_number": 187,
                    "sequence_number": 6031594,
                    "syslog_time": "2025-05-03T13:20:50.000Z",
-                    "time": "2025-05-03T03:20:48.739Z",
-                    "timezone": "AEST"
+                    "time": "2025-05-03T13:20:48.739+10:00",
+                    "timezone": "Australia/Sydney"


syslog_time is UTC but has the same time (ignoring the zone) as the time and @timestamp. This is incorrect.

The syslog_time can't be interpreted as UTC, since that timestamp is zone-less. Absent of zone info, either tz_offset needs to be used, or in this case, the zone provided by the more precise timestamp.

As of right now, syslog_time and @timestamp are 10 hours off, when they are only 2 seconds off from each other.

What we could do is set a "global" time zone (event.timezone) after we've done the necessary processing the of time zone and use that for all subsequent date processing.

taylor-swanson · 2025-07-16T13:55:07Z

packages/cisco_nexus/data_stream/log/_dev/test/pipeline/test-nexus.log-expected.json

                    "threshold_value": "-1.0 dBm",
-                    "time": "2025-06-14T01:34:35.000Z",
-                    "timezone": "AEST",
+                    "time": "2025-06-14T11:34:35.000Z",


Incorrect time zone

dwhyrock · 2025-07-17T15:39:52Z

@P1llus do you have a timeline as to when you'll address @taylor-swanson 's comments? It'd be nice to get this resolved to help the SDH customer.

resolve timestamp and timezone issues with cisco_nexus

b548756

P1llus added enhancement New feature or request Integration:cisco_nexus Cisco Nexus Team:Security-Deployment and Devices Deployment and Devices Security team [elastic/sec-deployment-and-devices] labels Jul 11, 2025

P1llus added 7 commits July 11, 2025 13:02

update all inputs to have tz_map available

d37ee7e

fix 2 missing brackets in grok patterns

2a7d01e

minor changes to not break preserve custom fields

d19e88c

skip output format for one date processor now that we don't need it

9378154

set syslog_time date processor later to not mess up to prevent settin…

56b5280

…g even more formats on other date processors

fix missing mapping for last grok pattern

97badd7

add workaround for preserve_custom_fields

ea471b5

dwhyrock mentioned this pull request Jul 11, 2025

[cisco_nexus] Fix timezone override bug #14458

Closed

5 tasks

P1llus added 2 commits July 11, 2025 14:26

update system test vars to include tz_map

648278b

update format of tz_* in agent configs

e98d50b

P1llus marked this pull request as ready for review July 11, 2025 12:40

P1llus requested a review from a team as a code owner July 11, 2025 12:40

P1llus added 2 commits July 11, 2025 15:14

fixing system tests and agent config typo

3712469

elastic package format

1d15d60

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 11, 2025

dwhyrock reviewed Jul 11, 2025

View reviewed changes

dwhyrock requested a review from a team July 15, 2025 13:08

taylor-swanson reviewed Jul 16, 2025

View reviewed changes

Merge branch 'main' into cisco_nexus_timestamp_cleanup

e5ba2c5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Cisco Nexus] Improve timezone and timestamp handling #14504

[Cisco Nexus] Improve timezone and timestamp handling #14504

P1llus commented Jul 11, 2025 •

edited

Loading

Uh oh!

elasticmachine commented Jul 11, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 11, 2025

Uh oh!

elastic-sonarqube bot commented Jul 11, 2025

Uh oh!

elasticmachine commented Jul 11, 2025

Uh oh!

dwhyrock left a comment

Uh oh!

dwhyrock Jul 11, 2025

Uh oh!

dwhyrock Jul 11, 2025

Uh oh!

dwhyrock Jul 11, 2025

Uh oh!

taylor-swanson Jul 11, 2025

Uh oh!

taylor-swanson Jul 16, 2025

Uh oh!

taylor-swanson Jul 16, 2025

Uh oh!

dwhyrock commented Jul 17, 2025

Uh oh!

Uh oh!

[Cisco Nexus] Improve timezone and timestamp handling #14504

Are you sure you want to change the base?

[Cisco Nexus] Improve timezone and timestamp handling #14504

Conversation

P1llus commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Uh oh!

elasticmachine commented Jul 11, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jul 11, 2025

🚀 Benchmarks report

Uh oh!

elastic-sonarqube bot commented Jul 11, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Jul 11, 2025

💚 Build Succeeded

History

Uh oh!

dwhyrock left a comment

Choose a reason for hiding this comment

Uh oh!

dwhyrock Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

dwhyrock Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

dwhyrock Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taylor-swanson Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

taylor-swanson Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

taylor-swanson Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

dwhyrock commented Jul 17, 2025

Uh oh!

Uh oh!

P1llus commented Jul 11, 2025 •

edited

Loading