Add more instrumentation #4555

ndr-ds · 2025-09-12T14:45:23Z

Motivation

Now that we have distributed tracing (after #4556), we need more instrumentation so we have data about more functions in the breakdowns.

Proposal

Instrument more functions with telemetry_only so that we don't get spammed in our logs, but the spans still get sent to Tempo.

Test Plan

Tested this with #4556, saw the spans properly show in the breakdowns.

Release Plan

Nothing to do / These changes follow the usual release cycle.

ndr-ds · 2025-09-12T14:45:34Z

Reconnect on max_age GoAway #4557
Use release_max_level_trace instead #4554
Tracing dashboard #4559
Add more instrumentation #4555 👈 (View in Graphite)
Adding Tempo for distributed tracing #4556
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

deuszx · 2025-09-18T08:25:36Z

linera-chain/src/chain.rs


+    #[instrument(target = "telemetry_only", skip_all, fields(
+        chain_id = %self.chain_id(),
+        next_block_height = %self.tip_state.get().next_block_height


I think the block height is unnecessary detail, it will obscure the trace.

deuszx · 2025-09-18T08:26:18Z

linera-chain/src/chain.rs


+    #[instrument(target = "telemetry_only", skip_all, fields(
+        chain_id = %self.chain_id(),
+        target = %target,


We should really decide how we label things. I've seen things like nickname , target, etc. and the are really NOT helping.

deuszx · 2025-09-18T08:26:57Z

linera-chain/src/chain.rs

    /// Removes the incoming message bundles in the block from the inboxes.
+    #[instrument(target = "telemetry_only", skip_all, fields(
+        chain_id = %self.chain_id(),
+        timestamp = %timestamp


I'd drop timestamp from there.

deuszx · 2025-09-18T08:27:21Z

linera-core/src/chain_worker/actor.rs


    /// Runs the worker until there are no more incoming requests.
    #[instrument(
+        target = "telemetry_only",


So we won't see this in the normal logs anymore?

Will remove

deuszx · 2025-09-18T08:28:07Z

linera-core/src/chain_worker/actor.rs

                            break; // Request sender was dropped.
                        };
-                        Box::pin(worker.handle_request(request).instrument(span)).await;
+                        Box::pin(worker.handle_request(request)).instrument(span).await;


Small change but are you sure this is what we want? @Twey has more experience with these spans.

If we instrument inside the pin, we reach the compilation recursion limit

deuszx · 2025-09-18T08:28:24Z

linera-core/src/chain_worker/state.rs


    /// Handles a request and applies it to the chain state.
-    #[instrument(skip_all)]
+    #[instrument(target = "telemetry_only", skip_all, fields(chain_id = %self.chain_id()))]


Same – this won't show up in the normal logs anymore?

deuszx · 2025-09-18T08:28:38Z

linera-core/src/chain_worker/state.rs

    /// Returns the requested blob, if it belongs to the current locking block or pending proposal.
+    #[instrument(target = "telemetry_only", skip_all, fields(
+        chain_id = %self.chain_id(),
+        blob_id = %blob_id


blob_id is IMHO unnecessary.

deuszx

I think we should discuss which logs we want for telemtry-only vs log file. Also, I think some details in the trace are unneccessary: timestamp, block height.

deuszx

I think a lot of logging, especially in grpc.rs was read in the console/log file. Now it's all telemetry-only.

ndr-ds · 2025-09-22T15:20:39Z

Ah, you're right! I'll do a pass, and if it was already INFO level before, the span should be sent already even without the telemetry_only thing. So I'll make sure that behavior is maintained for those before I merge

Now that we have distributed tracing (after #4556), we need more instrumentation so we have data about more functions in the breakdowns. Instrument more functions with `telemetry_only` so that we don't get spammed in our logs, but the spans still get sent to Tempo. Tested this with #4556, saw the spans properly show in the breakdowns. - Nothing to do / These changes follow the usual release cycle.

This was referenced Sep 12, 2025

Decrease block cache size, increase execution state cache size #4544

Merged

eBPF Pyroscope #4469

Merged

Optimize RocksDB prefix searches #4487

Merged

Continuous memory profiling #4498

Closed

This was referenced Sep 12, 2025

Use release_max_level_trace instead #4554

Open

Adding Tempo for distributed tracing #4556

Merged

Reconnect on max_age GoAway #4557

Draft

ndr-ds changed the base branch from 09-10-use_release_max_level_trace_instead to graphite-base/4555 September 12, 2025 16:26

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from e3a8454 to 599673a Compare September 12, 2025 16:26

ndr-ds changed the base branch from graphite-base/4555 to 09-09-adding_tempo_for_open_tracing September 12, 2025 16:26

ndr-ds mentioned this pull request Sep 12, 2025

Tracing dashboard #4559

Draft

ndr-ds changed the base branch from 09-09-adding_tempo_for_open_tracing to graphite-base/4555 September 12, 2025 19:29

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from 599673a to 45b67e5 Compare September 12, 2025 19:58

ndr-ds force-pushed the graphite-base/4555 branch from 1224256 to 4137625 Compare September 12, 2025 19:58

ndr-ds changed the base branch from graphite-base/4555 to 09-09-adding_tempo_for_open_tracing September 12, 2025 19:58

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from 45b67e5 to 1736f76 Compare September 12, 2025 20:04

ndr-ds force-pushed the 09-09-adding_tempo_for_open_tracing branch 2 times, most recently from 8125d96 to 0cc954c Compare September 12, 2025 20:11

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from 1736f76 to e700d67 Compare September 12, 2025 20:11

ndr-ds force-pushed the 09-09-adding_tempo_for_open_tracing branch from 0cc954c to 76f7ddd Compare September 12, 2025 20:46

ndr-ds force-pushed the 09-10-add_more_instrumentation branch 2 times, most recently from 7a6a2aa to 59b4fa0 Compare September 12, 2025 22:27

ndr-ds force-pushed the 09-09-adding_tempo_for_open_tracing branch from 76f7ddd to 7131de0 Compare September 12, 2025 22:27

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from 59b4fa0 to e46dc1a Compare September 12, 2025 22:57

ndr-ds force-pushed the 09-09-adding_tempo_for_open_tracing branch from 7131de0 to d6d672c Compare September 12, 2025 22:57

This was referenced Sep 15, 2025

Add feature to use jemalloc as memory allocator #4571

Merged

Continuous memory profiling #4572

Merged

Add memory profiling dashboard #4573

Merged

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from e46dc1a to f1302de Compare September 15, 2025 13:39

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from 3c906b7 to 3de7061 Compare September 18, 2025 01:28

ndr-ds requested review from afck, deuszx, eldios, ma2bd and Twey September 18, 2025 01:50

ndr-ds marked this pull request as ready for review September 18, 2025 01:50

deuszx reviewed Sep 18, 2025

View reviewed changes

ndr-ds force-pushed the 09-09-adding_tempo_for_open_tracing branch from f62b7d2 to cfa2f37 Compare September 22, 2025 13:50

ndr-ds force-pushed the 09-10-add_more_instrumentation branch 2 times, most recently from 82bf2c1 to a866aa0 Compare September 22, 2025 14:17

Base automatically changed from 09-09-adding_tempo_for_open_tracing to main September 22, 2025 15:07

deuszx approved these changes Sep 22, 2025

View reviewed changes

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from a866aa0 to b9e388c Compare September 22, 2025 17:12

Add more instrumentation

21cef69

ndr-ds force-pushed the 09-10-add_more_instrumentation branch from b9e388c to 21cef69 Compare September 22, 2025 17:24

ndr-ds added this pull request to the merge queue Sep 22, 2025

Merged via the queue into main with commit 0982bfd Sep 22, 2025
32 checks passed

ndr-ds deleted the 09-10-add_more_instrumentation branch September 22, 2025 20:38

ndr-ds mentioned this pull request Sep 22, 2025

Performance and tooling related backports #4626

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add more instrumentation #4555

Add more instrumentation #4555

Uh oh!

ndr-ds commented Sep 12, 2025 •

edited

Loading

Uh oh!

ndr-ds commented Sep 12, 2025 •

edited

Loading

Uh oh!

deuszx Sep 18, 2025

Uh oh!

deuszx Sep 18, 2025

Uh oh!

deuszx Sep 18, 2025

Uh oh!

deuszx Sep 18, 2025

Uh oh!

ndr-ds Sep 22, 2025

Uh oh!

deuszx Sep 18, 2025

Uh oh!

ndr-ds Sep 22, 2025

Uh oh!

deuszx Sep 18, 2025

Uh oh!

deuszx Sep 18, 2025

Uh oh!

deuszx left a comment

Uh oh!

deuszx left a comment

Uh oh!

ndr-ds commented Sep 22, 2025

Uh oh!

Uh oh!

Uh oh!

Add more instrumentation #4555

Add more instrumentation #4555

Uh oh!

Conversation

ndr-ds commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Proposal

Test Plan

Release Plan

Uh oh!

ndr-ds commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deuszx left a comment

Choose a reason for hiding this comment

Uh oh!

deuszx left a comment

Choose a reason for hiding this comment

Uh oh!

ndr-ds commented Sep 22, 2025

Uh oh!

Uh oh!

Uh oh!

ndr-ds commented Sep 12, 2025 •

edited

Loading

ndr-ds commented Sep 12, 2025 •

edited

Loading