Add imports section in semconv to support references on attribute_groups/metrics/events/entities from custom registries

.gitignore

@@ -40,4 +40,7 @@ output.json
 resolved-schema.yaml
 
 # Ignore the temporary repo dir created by `cargo xtask history` command
-history-temp-repo/
+history-temp-repo/
+
+# Exclude patch files
+*.patch

CHANGELOG.md

@@ -4,6 +4,8 @@ All notable changes to this project will be documented in this file.
 
 # [NEXT] - NEXT
 
+- Add `imports` section to semconv spec. Custom registries can now import groups
+  by name or by wildcard. ([#769](https://github.com/open-telemetry/weaver/pull/769/) by @lquerel)
 - Add support for metrics to `registry emit`
   ([#767](https://github.com/open-telemetry/weaver/pull/767) by @jerbly)
 - Adds `value_type` to metric: `int` or `double`. Data-point and exemplar values are live-checked.

README.md

@@ -7,7 +7,7 @@
 [![Slack](https://img.shields.io/badge/Slack-OpenTelemetry_Weaver-purple)](https://cloud-native.slack.com/archives/C0697EXNTL3)
 ----
 
-[Getting started](#getting-started) | [Main commands](#main-commands) | [Generate Doc & Code](crates/weaver_forge/README.md) | [Architecture](docs/architecture.md) | [Change log](CHANGELOG.md) | [Contributing](CONTRIBUTING.md) | [Links](#links) |
+[Getting started](#getting-started) | [Main commands](#main-commands) | [Generate Doc & Code](crates/weaver_forge/README.md) | [Define your registry](docs/define-your-own-telemetry-schema.md) | [Architecture](docs/architecture.md) | [Change log](CHANGELOG.md) | [Contributing](CONTRIBUTING.md) | [Links](#links) |
 
 > [!NOTE]
 > Codegen authors, please refer to the following documentation to learn how to
@@ -157,6 +157,8 @@ Telemetry Schemas.
 
 ## Documentation
 
+- [Define your own telemetry schema](docs/define-your-own-telemetry-schema.md): A guide on how to define your own
+  telemetry schema using semantic conventions.
 - [Weaver Architecture](docs/architecture.md): A document detailing the architecture of the project.
 - [Weaver Configuration](docs/weaver-config.md): A document detailing the configuration options available.
 - [Weaver Forge](crates/weaver_forge/README.md): An integrated template engine designed to generate

crates/weaver_resolver/Cargo.toml

@@ -24,7 +24,7 @@ serde_json.workspace = true
 walkdir.workspace = true
 miette.workspace = true
 itertools.workspace = true
-log.workspace = true
+globset.workspace = true
 
 [dev-dependencies]
 glob = "0.3.2"

crates/weaver_resolver/README.md

@@ -1,6 +1,7 @@
 # Telemetry Schema Resolution Process
 
 Resolution Process Status:
+
 - Semantic Convention Registry: **Fully Implemented**, **Partially Tested**
 - Application Telemetry Schema: **Partially Implemented**
 
@@ -31,18 +32,22 @@ structure during the parsing process.
 
 The resolution process for semantic conventions is a multistep process that
 involves the following steps:
+
 - Load all semantic conventions from the registry
 - Resolve iteratively all semantic conventions. This involves the maintenance
   of an unresolved semantic convention list and a resolved semantic convention
   list. The resolution process involves the following steps:
-  - Resolve iteratively all `extends` parent/child clauses until no more
-    resolvable `extends` are found. The extended entity inherits attributes
-    from the parent entity.
-  - Resolve iteratively all attributes `ref` until no more resolvable `ref` are
-    found.
+    - Resolve iteratively all `extends` parent/child clauses until no more
+      resolvable `extends` are found. The extended entity inherits attributes
+      from the parent entity.
+    - Resolve iteratively all attributes `ref` until no more resolvable `ref`are
+      found.
+- In the case of a registry importing another registry, the imports section is
+  used to import the metric, event, and entity groups specified by referencing
+  the names of these groups or by using a wildcard expression.
 - Validate the resolved semantic conventions
-  - No more unresolved `ref` or `extends` clauses. The unresolved list should
-    be empty.
+    - No more unresolved `ref` or `extends` clauses. The unresolved list should
+      be empty.
 
 ## Lineage (experimental)
 
@@ -58,6 +63,7 @@ The resolution process can optionally compute the lineage for each attribute of
 a semantic convention registry. The lineage as such is not part of the syntax
 and structure of a semantic convention; rather, it's an extension produced by
 the `weaver` tool, intended for use in scenarios such as:
+
 - A semconv author wishes to verify the exact path followed by the resolution
   process in the case of a complex cascade of inheritance across multiple levels
   between groups.
@@ -79,7 +85,9 @@ follows:
       "id": "<group id>",
       "type": "<group type>",
       "brief": "<group brief>",
-      "attributes": [ /* ... */ ],
+      "attributes": [
+        /* ... */
+      ],
       "lineage": {
         // `source_file` is either:
         // - a relative path in the case of a registry that has been explicitly
@@ -101,9 +109,15 @@ follows:
             // The field names inherited from the source group. This field is
             // present only if the attribute is inherited.
             // We assume all are inherited unless overridden.
-            "inherited_fields": [ "<field name>", /* ... */ ],
+            "inherited_fields": [
+              "<field name>",
+              /* ... */
+            ],
             // The field names overridden in the local group.
-            "locally_overridden_fields": [ "<field name>", /* ... */ ],
+            "locally_overridden_fields": [
+              "<field name>",
+              /* ... */
+            ],
           }
         }
       }

crates/weaver_resolver/data/multi-registry/custom_registry/custom_registry.yaml

@@ -30,3 +30,11 @@ groups:
         requirement_level: recommended
       - ref: error.type
         requirement_level: required
+
+imports:
+  metrics:
+    - example.*
+  entities:
+    - gcp.*
+  events:
+    - session.start

crates/weaver_resolver/data/multi-registry/otel_registry/otel_registry.yaml

@@ -21,3 +21,64 @@ groups:
         requirement_level: recommended
         examples:
           "Unused"
+
+  - id: metric.example.counter
+    type: metric
+    metric_name: example.counter
+    stability: development
+    brief: A counter of the number of messages processed.
+    instrument: counter
+    unit: "1"
+    value_type: int
+    attributes:
+      - ref: error.type
+
+  - id: entity.gcp.apphub.application
+    type: entity
+    stability: development
+    name: gcp.apphub.application
+    brief: >
+      Attributes denoting data from an Application in AppHub.
+      See [AppHub overview](https://cloud.google.com/app-hub/docs/overview).
+    attributes:
+      - ref: error.type
+  - id: entity.gcp.apphub.service
+    type: entity
+    stability: development
+    name: gcp.apphub.service
+    brief: >
+      Attributes denoting data from a Service in AppHub.
+      See [AppHub overview](https://cloud.google.com/app-hub/docs/overview).
+    attributes:
+      - ref: error.type
+
+  - id: event.session.start
+    stability: development
+    type: event
+    name: session.start
+    brief: >
+      Indicates that a new session has been started, optionally linking to the prior session.
+    note: >
+      For instrumentation that tracks user behavior during user sessions, a `session.start` event MUST be emitted
+      every time a session is created. When a new session is created as a continuation of a prior session,
+      the `session.previous_id` SHOULD be included in the event. The values of `session.id` and `session.previous_id`
+      MUST be different.
+
+      When the `session.start` event contains both `session.id` and `session.previous_id` fields, the event indicates
+      that the previous session has ended. If the session ID in `session.previous_id` has not yet ended via explicit
+      `session.end` event, then the consumer SHOULD treat this continuation event as semantically equivalent to
+      `session.end(session.previous_id)` and `session.start(session.id)`.
+    attributes:
+      - ref: error.type
+  - id: event.session.end
+    stability: development
+    type: event
+    name: session.end
+    brief: >
+      Indicates that a session has ended.
+    note: >
+      For instrumentation that tracks user behavior during user sessions, a `session.end` event SHOULD be emitted
+      every time a session ends. When a session ends and continues as a new session, this event SHOULD be
+      emitted prior to the `session.start` event.
+    attributes:
+      - ref: error.type

crates/weaver_resolver/src/lib.rs

@@ -159,6 +159,16 @@ pub enum Error {
         attribute_id: String,
     },
 
+    /// Invalid import wildcard.
+    #[error("Invalid import wildcard: {error:?}")]
+    #[diagnostic(help(
+        "Check the wildcard syntax supported here: https://crates.io/crates/globset"
+    ))]
+    InvalidWildcard {
+        /// The error that occurred.
+        error: String,
+    },
+
     /// A container for multiple errors.
     #[error("{:?}", format_errors(.0))]
     CompoundError(#[related] Vec<Error>),
@@ -438,15 +448,53 @@ mod tests {
                         assert_eq!(semconv_spec.groups().len(), 2);
                         assert_eq!(&semconv_spec.groups()[0].id, "shared.attributes");
                         assert_eq!(&semconv_spec.groups()[1].id, "metric.auction.bid.count");
+                        assert_eq!(
+                            semconv_spec
+                                .imports()
+                                .unwrap()
+                                .metrics
+                                .as_ref()
+                                .unwrap()
+                                .len(),
+                            1
+                        );
+                        assert_eq!(
+                            semconv_spec
+                                .imports()
+                                .unwrap()
+                                .events
+                                .as_ref()
+                                .unwrap()
+                                .len(),
+                            1
+                        );
+                        assert_eq!(
+                            semconv_spec
+                                .imports()
+                                .unwrap()
+                                .entities
+                                .as_ref()
+                                .unwrap()
+                                .len(),
+                            1
+                        );
                     }
                     "otel" => {
                         assert_eq!(
                             source.path,
                             "data/multi-registry/otel_registry/otel_registry.yaml"
                         );
-                        assert_eq!(semconv_spec.groups().len(), 2);
+                        assert_eq!(semconv_spec.groups().len(), 7);
                         assert_eq!(&semconv_spec.groups()[0].id, "otel.registry");
                         assert_eq!(&semconv_spec.groups()[1].id, "otel.unused");
+                        assert_eq!(&semconv_spec.groups()[2].id, "metric.example.counter");
+                        assert_eq!(
+                            &semconv_spec.groups()[3].id,
+                            "entity.gcp.apphub.application"
+                        );
+                        assert_eq!(&semconv_spec.groups()[4].id, "entity.gcp.apphub.service");
+                        assert_eq!(&semconv_spec.groups()[5].id, "event.session.start");
+                        assert_eq!(&semconv_spec.groups()[6].id, "event.session.end");
                     }
                     _ => panic!("Unexpected registry id: {}", source.registry_id),
                 }
@@ -463,10 +511,37 @@ mod tests {
                         // The group `otel.unused` shouldn't be garbage collected
                         let group = resolved_registry.group("otel.unused");
                         assert!(group.is_some());
+
+                        // These groups are referenced in the `imports` and should not be garbage
+                        // collected
+                        let group = resolved_registry.group("metric.example.counter");
+                        assert!(group.is_some());
+                        let group = resolved_registry.group("entity.gcp.apphub.application");
+                        assert!(group.is_some());
+                        let group = resolved_registry.group("entity.gcp.apphub.service");
+                        assert!(group.is_some());
+                        let group = resolved_registry.group("event.session.start");
+                        assert!(group.is_some());
+                        let group = resolved_registry.group("event.session.end");
+                        assert!(group.is_some());
                     } else {
-                        // The group `otel.unused` should be garbage collected
+                        // These groups should be garbage collected because they are not referenced
+                        // anywhere (in ref or imports)
                         let group = resolved_registry.group("otel.unused");
                         assert!(group.is_none());
+                        let group = resolved_registry.group("event.session.end");
+                        assert!(group.is_none());
+
+                        // These groups are referenced in the `imports` and should not be garbage
+                        // collected
+                        let group = resolved_registry.group("metric.example.counter");
+                        assert!(group.is_some());
+                        let group = resolved_registry.group("entity.gcp.apphub.application");
+                        assert!(group.is_some());
+                        let group = resolved_registry.group("entity.gcp.apphub.service");
+                        assert!(group.is_some());
+                        let group = resolved_registry.group("event.session.start");
+                        assert!(group.is_some());
                     }
 
                     let metrics = resolved_registry.groups(GroupType::Metric);