Using Prometheus 3 OpenTelemetry Endpoint to Ingest OTLP Metrics in Kubernetes

This blog post demonstrates how to send OTLP metrics to Prometheus 3 in a Kubernetes platform with both the Prometheus and OpenTelemetry Operators already installed.

Prerequisites

Ensure you have the following components installed in your Kubernetes cluster:

• Prometheus 3 (with the OpenTelemetry ingestion endpoint enabled)
• OpenTelemetry Operator (for managing OpenTelemetry Collector instances)
• A sample application that generates OTLP metrics

Deploying an OpenTelemetry Collector to Export Metrics

We’ll configure an OpenTelemetry Collector to scrape application metrics and forward them to Prometheus 3.

OpenTelemetry Collector Configuration in CRD

Instead of using a ConfigMap, we embed the configuration directly in the OpenTelemetryCollector custom resource:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:

    processors:
      batch:

    exporters:
      prometheusremotewrite:
        endpoint: "http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/otlp"

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheusremotewrite]

Configuring Prometheus 3 to Accept OTLP Metrics via the Prometheus Operator

To enable the OTLP endpoint in Prometheus using the Prometheus Operator, modify the Prometheus custom resource:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  containers:
    - name: prometheus
      args:
        - "--web.enable-remote-write-receiver"
        - "--enable-feature=otlp-write-receiver"
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  podMonitorSelector: {}
  resources:
    requests:
      memory: 400Mi

These flags enable the OTLP metrics endpoint at /api/v1/otlp.

Deploying a Sample Application with OTLP Metrics

We will deploy a simple sample application with OpenTelemetry instrumentation to send metrics to the OpenTelemetry Collector.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: sample-app
          image: myorg/sample-app:latest
          ports:
            - containerPort: 8080
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector.monitoring.svc.cluster.local:4317"

Verifying Metrics in Prometheus

After deploying the sample application, verify that metrics are successfully flowing into Prometheus:

• Access the Prometheus UI:

kubectl port-forward svc/prometheus-operated 9090 -n monitoring

• Navigate to http://localhost:9090 and use the Prometheus expression browser to check for incoming OTLP metrics.

Conclusion

With Prometheus 3’s OpenTelemetry endpoint, integrating OTLP metrics into a Kubernetes-based observability stack becomes much simpler. This setup allows teams to leverage OpenTelemetry’s powerful telemetry collection capabilities while continuing to use Prometheus as a reliable backend for metrics storage and querying.