> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abbyy.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring with Prometheus

> Collect Vantage 3.0 self-hosted metrics with Prometheus across Istio and Linkerd service meshes.

This guide sets up a self-managed Prometheus to collect Vantage 3.0 self-hosted metrics in any environment. On AKS, where Azure Monitor Workspace and Azure Managed Grafana are available, see [Azure Managed Monitoring](/vantage/self-hosted/v3.0/monitoring/azure-managed) for a managed-service deployment of the same pattern.

<Note>
  The examples use [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), but the approach is generic and applies to any Prometheus deployment. For the mesh-specific constraints, see [Monitoring Overview](/vantage/self-hosted/v3.0/monitoring/overview).
</Note>

## How Vantage exposes metrics

Vantage application pods are instrumented with OpenTelemetry SDKs and expose Prometheus-format metrics on `/metrics-text` at container port 8080. Services that expose this endpoint carry the label `abbyy.platform.metrics.text.endpoint: "1"` and map a service port (typically 80) to the container port with the port name `http`. Prometheus discovers and scrapes these services via a `ServiceMonitor`; no separate OpenTelemetry Collector is required for metrics collection.

For the full concept and service-mesh constraints, see [Monitoring Overview](/vantage/self-hosted/v3.0/monitoring/overview).

## Prerequisites

* A running Vantage installation (operator and workloads deployed).
* Helm 3.
* `kubectl` access to the cluster.
* When KEDA is enabled, a Prometheus instance reachable as `prometheus-operated.observability.svc.cluster.local:9090`.

## Configure Prometheus

If you plan to ingest OTLP metrics (see [OTLP ingestion](#otlp-ingestion)), enable the `otlp-write-receiver` feature and pass the `--web.enable-otlp-receiver` argument to the Prometheus instance.

Add the Helm repo:

```bash theme={null}
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
```

Create a values file (`observability-values.yaml`):

```yaml theme={null}
prometheus:
  prometheusSpec:
    enableFeatures:
      - otlp-write-receiver
    additionalArgs:
      - name: web.enable-otlp-receiver
        value: ""
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: "2"
        memory: 4Gi
grafana:
  enabled: true
  adminPassword: "changeme"  # Override in production
  persistence:
    enabled: true
    size: 10Gi
alertmanager:
  enabled: false
nodeExporter:
  enabled: true
kube-state-metrics:
  enabled: true
# Managed Kubernetes (e.g. AKS, EKS, GKE) does not expose control plane
# endpoints. Disable these to avoid scrape errors.
kubeApiServer:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false
```

Install:

```bash theme={null}
helm install observability prometheus-community/kube-prometheus-stack \
  -n observability --create-namespace \
  -f observability-values.yaml
```

Verify the pods come up:

```bash theme={null}
kubectl get pods -n observability
```

You should see Prometheus, Grafana, the operator, kube-state-metrics, and node-exporter pods running. Node-exporter runs as a DaemonSet; pending pods are normal if nodes are still scaling.

## Configure Prometheus to scrape Vantage

The `vantage-selfhosted` chart can create the requisite `ServiceMonitor` objects to monitor Vantage:

```bash theme={null}
helm upgrade --install abbyy-vantage \
  # ...
  --set observability.prometheus.serviceMonitor.create=true
```

See the chart documentation for more options.

The `observability` key is top-level and must not be nested under `vantage`.

## Linkerd mTLS

With Linkerd's default permissive policy, Prometheus can normally scrape the Linkerd-proxied Vantage endpoints over plaintext. Install Prometheus in the `observability` namespace, enable the Vantage `ServiceMonitor`, and confirm every Vantage target is `UP`.

If a strict Linkerd authorization policy blocks the scrape, use one of these approaches:

1. Enable Linkerd injection for the `observability` namespace so Prometheus participates in the mesh.
2. Configure `observability.prometheus.serviceMonitor.scheme: https` and a `tlsConfig` that references the client certificates made available to Prometheus by your Linkerd configuration.

Do not enable KEDA until Prometheus returns current values for the Vantage metrics used by the `ScaledObject` resources. See [Autoscaling with KEDA](/vantage/self-hosted/v3.0/performance/autoscaling) for the inspection queries and readiness checks.

## Istio mTLS

When Vantage is placed in an Istio mesh using STRICT mTLS (the default when `ingress.enabled` is `true` and `ingress.type` is `istio`), Prometheus and the `ServiceMonitor` must be configured to scrape endpoints over mTLS, because Prometheus uses pod IPs directly rather than the service name.

Configure the Prometheus pod(s) with the following annotations:

```yaml theme={null}
annotations:
  proxy.istio.io/config: |
    proxyMetadata:
      OUTPUT_CERTS: /etc/istio-output-certs
  sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'
```

and the following `volume` and `volumeMount` (names and paths are customizable; the examples below use these):

```yaml theme={null}
volumes:
  - name: istio-certs
    emptyDir:
      medium: Memory
volumeMounts:
  - name: istio-certs
    mountPath: /etc/prom-certs/
    readOnly: true
```

With kube-prometheus-stack, this can be done with the following values:

```yaml theme={null}
prometheus:
  prometheusSpec:
    podMetadata:
      annotations:
        # force Istio to generate certificates into a local volume directory
        "proxy.istio.io/config": |
          proxyMetadata:
            OUTPUT_CERTS: /etc/istio-output-certs
        "sidecar.istio.io/userVolumeMount": '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'
    # shared volume where Istio will write certificates
    volumes:
      - name: istio-certs
        emptyDir: {}
    volumeMounts:
      - name: istio-certs
        mountPath: /etc/prom-certs
```

Alternatively, see the [official Istio docs](https://istio.io/latest/docs/ops/integrations/prometheus/#tls-settings).

Next, install the `vantage-selfhosted` chart so the `ServiceMonitor`(s) are configured to use these certificates:

```yaml theme={null}
# values.vantage-selfhosted.yaml
observability:
  prometheus:
    serviceMonitor:
      scheme: https
      tlsConfig:
        caFile: /etc/prom-certs/root-cert.pem
        certFile: /etc/prom-certs/cert-chain.pem
        keyFile: /etc/prom-certs/key.pem
        insecureSkipVerify: true  # Prometheus does not support Istio security naming, thus skip verifying target pod certificate
```

## Verify metrics are flowing

Port-forward to the Prometheus UI:

```bash theme={null}
kubectl port-forward -n observability svc/observability-kube-prometheus-prometheus 9090:9090
```

Check that Vantage targets are discovered and healthy:

```bash theme={null}
curl -s http://localhost:9090/api/v1/targets \
  | jq '[.data.activeTargets[] | select(.scrapePool // "" | contains("vantage"))] | group_by(.health) | map({health: .[0].health, count: length})'
```

Query for a Vantage application metric:

```bash theme={null}
curl -s "http://localhost:9090/api/v1/query?query=process_cpu_count" \
  | jq '.data.result | length'
```

A non-zero result confirms metrics are being scraped. You can also browse targets at `http://localhost:9090/targets` to see individual scrape status.

## Access Grafana

```bash theme={null}
kubectl port-forward -n observability svc/observability-grafana 3000:80
```

Open `http://localhost:3000`. Default credentials are `admin` / the password set in the values file. Prometheus is pre-configured as a data source.

## Available metrics

Vantage applications expose process, HTTP client/server, and ABBYY platform metrics through the OpenTelemetry SDK, all labeled with `service`, `pod`, `namespace`, and `otel_scope_name`. See [Available metrics](/vantage/self-hosted/v3.0/monitoring/overview#available-metrics) for the full catalog.

## OTLP ingestion

<Note>
  OTLP ingestion is push-based and independent of the scrape-based `ServiceMonitor` flow above. Use it only for services that push OpenTelemetry metrics rather than exposing a scrape endpoint.
</Note>

When Prometheus is configured with the OTLP receiver enabled (`--web.enable-otlp-receiver`), any service that pushes OTLP metrics via HTTP can send them to:

```text theme={null}
http://observability-kube-prometheus-prometheus.observability:9090/api/v1/otlp/v1/metrics
```