Telemetry gateway

Overview

The telemetry gateway is an OpenTelemetry Collector component in the SUSE® Observability Agent that accepts metrics and traces pushed over OTLP from application-embedded OpenTelemetry SDKs. It enriches the data with Kubernetes metadata and forwards it to the SUSE® Observability platform.

Use the telemetry gateway when your workloads are instrumented with an OTel SDK (Java, Go, Python, Node.js, etc.) and export telemetry using OTLP. It is the recommended integration path for natively instrumented services.

If your workloads expose a Prometheus / OpenMetrics endpoint instead, see Scraping OpenMetrics with ServiceMonitor and PodMonitor.

Enable

The telemetry gateway is disabled by default. Enable it in your agent Helm values alongside the OTel master switch:

otel:
  enabled: true
  telemetryGateway:
    enabled: true

After the Helm upgrade, a Deployment named suse-observability-agent-otel-telemetry-gateway is created in the agent namespace, along with a ClusterIP Service, ClusterRole, and ServiceAccount.

Configure your SDKs

Point your application SDKs at the telemetry gateway Service. By default, the Service is suse-observability-agent-otel-telemetry-gateway in the suse-observability-agent namespace.

Set the following environment variables on each application pod. Most OTel SDKs pick these up automatically at startup without any code changes.

Log ingestion via OTLP is not yet supported. Set OTEL_LOGS_EXPORTER=none to avoid exporting logs. If a workload sends OTLP logs to the gateway anyway, they are written to the gateway pod logs for debug visibility and are not ingested as SUSE® Observability logs.

gRPC (port 4317)

OTEL_EXPORTER_OTLP_ENDPOINT=http://suse-observability-agent-otel-telemetry-gateway.suse-observability-agent.svc.cluster.local:4317
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=none
OTEL_SERVICE_NAME=<your-service-name>
OTEL_RESOURCE_ATTRIBUTES=k8s.pod.uid=$(POD_UID)

HTTP (port 4318)

OTEL_EXPORTER_OTLP_ENDPOINT=http://suse-observability-agent-otel-telemetry-gateway.suse-observability-agent.svc.cluster.local:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=none
OTEL_SERVICE_NAME=<your-service-name>
OTEL_RESOURCE_ATTRIBUTES=k8s.pod.uid=$(POD_UID)

Inject POD_UID via the Kubernetes downward API:

env:
  - name: POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid

Attribute ownership (security enforcement)

The collector overwrites the following attributes on every received payload, regardless of what the client sends. These attributes are used for RBAC and tenancy decisions, so the collector queries the Kubernetes API for authoritative values instead of relying on application-provided values.

  • k8s.cluster.name — set from the agent Helm release configuration

  • k8s.namespace.name — set from the Kubernetes API using the source pod’s identity

Attributes that applications must set:

  • service.name — the logical name of your service

  • service.version — the deployed version

  • deployment.environment — e.g. production or staging

  • k8s.pod.uid — recommended; inject via the downward API (see above) for reliable pod association. The collector also falls back to k8s.pod.ip and connection source address, but downward API injection is the most reliable method and should be preferred.

Span metrics

The gateway automatically generates span metrics (request rate, error rate, duration histograms) from incoming traces using the span_metrics connector. The generated metrics use the otel_span namespace and follow OTel semantic conventions:

  • Attribute span.name

  • Histogram name otel_span.duration

  • Counter without _total suffix

These metrics are forwarded to SUSE® Observability alongside your application metrics and can be used to build golden-signal dashboards without any additional instrumentation.

Operational monitoring

The gateway exposes standard OpenTelemetry Collector self-metrics on its internal Prometheus endpoint (:8888). These include signals such as exporter failures, refused data, process memory, and pipeline throughput. The gateway scrapes its own endpoint with a built-in prometheus/self receiver and forwards those self-metrics to the platform for operational visibility.

Network policies

The chart does not install NetworkPolicy resources. If your cluster enforces network policies, configure:

  • Ingress to the agent namespace on port 4317 (gRPC) or 4318 (HTTP) from application namespaces.

  • Egress from application pods to the agent namespace.

To allow a specific namespace to send telemetry to the gateway:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-otlp-to-gateway
  namespace: suse-observability-agent
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: otel-telemetry-gateway
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: my-app-namespace
      ports:
        - port: 4317
        - port: 4318

To block a namespace from sending telemetry, deny its egress to the gateway namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-otlp-egress
  namespace: restricted-namespace
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchExpressions:
              - key: kubernetes.io/metadata.name
                operator: NotIn
                values:
                  - suse-observability-agent

Scaling

The default replicaCount: 1 is suitable for most clusters. For high availability:

otel:
  telemetryGateway:
    replicaCount: 2

At two or more replicas, a PodDisruptionBudget is automatically created with maxUnavailable: 1, ensuring at least one pod remains available during voluntary disruptions.

Resource limits and requests can be tuned in the Helm values:

otel:
  telemetryGateway:
    resources:
      requests:
        cpu: 250m
        memory: 1Gi
      limits:
        cpu: 1
        memory: 2Gi

Overload protection

The collector includes memory_limiter as the first processor in every pipeline. It protects the process from OOM by rejecting data once memory usage exceeds 80% (with a 10% spike allowance). Under sustained overload, data is dropped to preserve collector stability — this is intentional behaviour. Per-producer rate limiting, sampling, and autoscaling are separate stories.

Span-metric generation also has a cardinality guard. otel.telemetryGateway.spanMetrics.aggregationCardinalityLimit limits the number of unique span-metric aggregation series held by the gateway. The default protects memory from high-cardinality spans, while larger clusters can tune it when needed:

otel:
  telemetryGateway:
    spanMetrics:
      aggregationCardinalityLimit: 5000

Product integration contract

This section is for SUSE product teams that want their Helm charts to export push-based OTLP telemetry through the telemetry gateway when the SUSE® Observability Agent supports it.

Capability detection

Before enabling auto-export by default, check for the marker CRD in Helm:

{{- if .Capabilities.APIVersions.Has "observability.suse.com/v1/suseobservabilityagents" }}

The agent installs this CRD when at least one product-facing OTel integration path is active, such as Prometheus scraping or the telemetry gateway. Treat it as a capability signal only. It does not identify the agent namespace, Service name, protocol, port, or NetworkPolicy reachability.

Endpoint

The default gateway Service name is suse-observability-agent-otel-telemetry-gateway: the agent Helm release name, suse-observability-agent, plus the -otel-telemetry-gateway suffix. Product charts should construct this endpoint from configurable release name, namespace, protocol, and port values so custom agent installations are supported.

Opt-in/override

Product charts should keep telemetry export configurable even when the marker CRD exists. Allow users to disable export, override the endpoint, and choose OTLP gRPC or HTTP/protobuf where the product supports both.

Attribute ownership

The collector owns k8s.cluster.name and k8s.namespace.name (see Attribute ownership). Products must not set or rely on user-supplied values for these. Products must set service.name, service.version, and deployment.environment. Products should inject k8s.pod.uid via the downward API — this is the most reliable pod-association method; the collector falls back to pod IP and connection source address.

Log ingestion

Log signals are not yet supported for SUSE® Observability log ingestion. Products must not export logs to the gateway at this time; any logs sent to the gateway are written to the gateway pod logs for debug visibility only.

See also