Scraping OpenMetrics with ServiceMonitor and PodMonitor

Overview

The SUSE Observability Agent bundles an OpenTelemetry Collector and a Target Allocator that scrape any Kubernetes workload exposing a Prometheus / OpenMetrics endpoint. The Target Allocator watches the ServiceMonitor and PodMonitor Custom Resources defined by the Prometheus Operator, distributes the scrape targets across collector pods, and ships the collected metrics over OTLP to SUSE Observability.

In practice this means you can:

  • Reuse existing ServiceMonitor and PodMonitor manifests written for the Prometheus Operator — no per-pod Agent annotations needed.

  • Run alongside an existing Prometheus Operator install, or use the SUSE Observability Agent as a drop-in replacement.

  • Opt workloads in or out with Kubernetes labels.

Step 1 — Enable OTel and Prometheus scraping on the Agent

First, install the SUSE Observability Agent on your cluster by following the quick start guide.

When you install (or upgrade) the agent Helm chart, set these values to turn on the OpenTelemetry Collector pipeline and the Prometheus scraping sub-pipeline:

otel:
  enabled: true
  prometheusScraping:
    monitorCrds:
      # Leave at the default, false, if the SerivceMonitor and PodMonitor CRDs
      # are already managed by another chart (for example kube-prometheus-stack) or
      # if you want to install them manually.
      enabled: true

Apply the values:

helm upgrade --install suse-observability-agent suse-observability/suse-observability-agent \
  --namespace suse-observability-agent \
  -f agent-values.yaml

This deploys an OpenTelemetry Collector and a Target Allocator alongside the rest of the agent.

Step 2 — Define a ServiceMonitor or PodMonitor

By default the Target Allocator only picks up monitors that carry the observability.suse.com/agent: scrape label. This keeps the agent from accidentally scraping every monitor in a cluster that already runs the Prometheus Operator.

A minimal ServiceMonitor for an HTTP /metrics endpoint behind a Service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-metrics
  namespace: my-app
  labels:
    observability.suse.com/agent: scrape
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app
  endpoints:
    - port: metrics
      path: /metrics

A PodMonitor for pods that aren’t fronted by a Service:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-app-metrics
  namespace: my-app
  labels:
    observability.suse.com/agent: scrape
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app
  podMetricsEndpoints:
    - port: metrics
      path: /metrics

For the full set of fields supported by these resources — scrape interval, relabeling, TLS, multiple endpoints, and so on — see the upstream Prometheus Operator reference:

To change which label the Target Allocator selects on, set otel.prometheusScraping.targetAllocator.prometheusCR.serviceMonitorSelector and podMonitorSelector on the agent Helm release.

Step 3 — Authenticated endpoints

If your /metrics endpoint requires authentication, reference a Kubernetes Secret from the monitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-metrics
  namespace: my-app
  labels:
    observability.suse.com/agent: scrape
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app
  endpoints:
    - port: metrics
      path: /metrics
      authorization:
        # type defaults to Bearer; set to Basic or another scheme if needed
        credentials:
          name: my-app-metrics-auth
          key: token

The Target Allocator does not have cluster-wide secret access, so two extra steps are required:

  1. List the secret’s namespace in otel.prometheusScraping.targetAllocator.prometheusCR.secretNamespaces.

  2. Deploy a Role and RoleBinding in that namespace granting the agent’s ServiceAccount (which lives in the release namespace) read access to secrets.

By default the Target Allocator also will not serve those secrets to the collector pods. Choose one of the options below to allow it:

  • Recommended — mTLS: enable mutual TLS between the Target Allocator and collectors so secrets travel encrypted. Requires cert-manager to be installed:

    otel:
  prometheusScraping:
    targetAllocator:
      mtlsEnabled: true

  • Only for isolated clusters — plain HTTP: allow secrets to be served over plain HTTP between agent components. Credentials travel unencrypted inside the cluster:

    otel:
  prometheusScraping:
    targetAllocator:
      allowInsecureAuthSecrets: true

Step 4 — Browse the collected metrics

Once the agent, collectors and your ServiceMonitor / PodMonitor are in place, metrics start flowing into SUSE Observability within a scrape interval. Find them in the telemetry inspector and bind them to topology components with metric bindings.

Product integration contract

This section is for SUSE product teams that want their Helm charts to expose Prometheus / OpenMetrics endpoints to the SUSE Observability Agent.

Capability detection

Before rendering SUSE Observability-specific ServiceMonitor or PodMonitor resources by default, check for the marker CRD in Helm:

{{- if .Capabilities.APIVersions.Has "observability.suse.com/v1/suseobservabilityagents" }}

The agent installs this CRD when at least one product-facing OTel integration path is active, such as Prometheus scraping or the telemetry gateway. Treat it as a capability signal only. It does not prove that a specific monitor selector, namespace selector, or NetworkPolicy allows the scrape.

Monitor labels

By default, add observability.suse.com/agent: scrape to ServiceMonitor and PodMonitor resources that should be picked up by the agent Target Allocator. Product charts should make this label configurable because agent administrators can change otel.prometheusScraping.targetAllocator.prometheusCR.serviceMonitorSelector and podMonitorSelector.

Opt-in/override

Product charts should keep monitor rendering configurable even when the marker CRD exists. Allow users to disable the SUSE Observability monitor resources, override labels, and configure scrape endpoint details such as port, path, interval, TLS, and authentication.

See also