Sample curl for posting a metric to http_server source

[tbenade]

Question

Hi, what would a curl command look like that posts a metric event to a Vector aggregator http_server source

type: http_server
address: "0.0.0.0:8080"
strict_path: false
encoding: "json"
path_key: "vector_http_path"

that is piped into a route with a route type of is_metric

type: route
inputs:
  - http_server
reroute_unmatched: true
route:
  logs:
    type: is_log
  metrics:
    type: is_metric
  traces:
    type: is_trace

into sink type datadog_metrics

I cant find anywhere what the payload should look like for me to test this path. Thanks!

Vector Config

No response

Vector Logs

No response

[pront]

Hi @tbenade, here is an example using the native proto to send multiple events in one shot:

Vector Config

sources:
  http_server:
    type: http_server
    address: "0.0.0.0:8080"
    strict_path: false
    decoding:
      codec: "native"
    framing:
      method: "bytes"


transforms:
  route:
    type: route
    inputs:
      - http_server
    reroute_unmatched: false
    route:
      logs:
        type: is_log
      metrics:
        type: is_metric
      traces:
        type: is_trace

sinks:
  console_for_metrics:
    inputs:
     - route.metrics
    type: console
    encoding:
      codec: "json"
      json:
        pretty: true

  console_for_logs:
    inputs:
      - route.logs
    type: console
    encoding:
      codec: json
      json:
        pretty: true

Generate Vector event proto

import requests
import event_pb2
from google.protobuf.timestamp_pb2 import Timestamp
from datetime import datetime, timezone

# Build a Metric
ts = Timestamp()
ts.FromDatetime(datetime.now(timezone.utc))

metric = event_pb2.Metric(
    name="first",
    timestamp=ts,
    kind=event_pb2.Metric.Absolute,
    gauge=event_pb2.Gauge(value=1.0),
    tags_v1={"service": "foo" }
)

metric2 = event_pb2.Metric(
    name="second",
    timestamp=ts,
    kind=event_pb2.Metric.Absolute,
    gauge=event_pb2.Gauge(value=2.0),
    tags_v1={"service": "bar" }
)

# Wrap into EventArray -> MetricArray
wrapper = event_pb2.EventArray(
    metrics=event_pb2.MetricArray(metrics=[metric, metric2])
)

# Serialize to protobuf wire format
data = wrapper.SerializeToString()

# Send raw bytes
resp = requests.post(
    "http://127.0.0.1:8080/",
    headers={"Content-Type": "application/x-protobuf"},
    data=data,
)
print(resp.status_code, resp.text)

You can send a batch of metrics at a time.

This config/method should work for all data types (logs, metrics, traces).

P.S. The encoding field should be avoided (will add this to the deprecations list).

[tbenade]

Thank you I will give it a go. @pront please could you expand on the 'encoding' deprecation? As I was just about to ask you a related question. If I may I might hijack this thread. We very sporadically get the following error which is pretty much the only error in our entire pipeline. The source of the events comes from a forwarder we have written that bundles, compresses and forwards billions of events a day out of AWS Cloudwatch. On failure we save the payload to a queue for inspection. To date we have found nothing peculiar about the events and all other code be it Javascript, Typescript or even some command-line tooling has not failed to uncompress and parse the payload. Was hoping you could offer an insights into what ghosts we are chasing thank you.

{
  "metadata": {
    "level": "ERROR",
    "kind": "event",
    "module_path": "vector::internal_events::codecs",
    "target": "vector::internal_events::codecs"
  },
  "hostname": "ip-x-x-x-x.ap-southeast-2.compute.internal",
  "stage": "processing",
  "service": "vector-aggregator",
  "error_type": "parser_failed",
  "internal_log_rate_limit": true,
  "pid": 1,
  "error_code": "decoder_deserialize",
  "vector": {
    "component_type": "http_server",
    "component_id": "http_server",
    "component_kind": "source"
  },
  "timestamp": 1756164877029,
  "error": "Error parsing JSON: Error(\"unexpected end of hex escape\", line: 1, column: 141329)"
}

[pront]

Hi @tbenade, nothing stands out. But feel free to create a new discussion and we can revisit.

[tbenade]

will do, but could you please expand on the 'encoding' deprecation comment? As it is still being actively promoted in documentation https://vector.dev/docs/reference/configuration/sources/http_server/#examples-application/json

[pront]

Yes, per https://github.com/vectordotdev/vector/blob/master/src/sources/http_server.rs#L230:

Using `encoding` is deprecated and does not have any effect when `decoding` or `framing` is provided. Configure `framing` and `decoding` instead.

All source decode payloads, so having an encoding setting at the source is confusing. Use decoding and framing instead.