Monitoring with Prometheus - ABBYY Documentation

This guide sets up a self-managed Prometheus to collect Vantage 3.0 self-hosted metrics in any environment. On AKS, where Azure Monitor Workspace and Azure Managed Grafana are available, see Azure Managed Monitoring for a managed-service deployment of the same pattern.

The examples use kube-prometheus-stack, but the approach is generic and applies to any Prometheus deployment. For the mesh-specific constraints, see Monitoring Overview.

How Vantage exposes metrics

Vantage application pods are instrumented with OpenTelemetry SDKs and expose Prometheus-format metrics on /metrics-text at container port 8080. Services that expose this endpoint carry the label abbyy.platform.metrics.text.endpoint: "1" and map a service port (typically 80) to the container port with the port name http. Prometheus discovers and scrapes these services via a ServiceMonitor; no separate OpenTelemetry Collector is required for metrics collection. For the full concept and service-mesh constraints, see Monitoring Overview.

Prerequisites

A running Vantage installation (operator and workloads deployed).
Helm 3.
kubectl access to the cluster.
When KEDA is enabled, a Prometheus instance reachable as prometheus-operated.observability.svc.cluster.local:9090.

Configure Prometheus

If you plan to ingest OTLP metrics (see OTLP ingestion), enable the otlp-write-receiver feature and pass the --web.enable-otlp-receiver argument to the Prometheus instance. Add the Helm repo:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create a values file (observability-values.yaml):

prometheus:
  prometheusSpec:
    enableFeatures:
      - otlp-write-receiver
    additionalArgs:
      - name: web.enable-otlp-receiver
        value: ""
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: "2"
        memory: 4Gi
grafana:
  enabled: true
  adminPassword: "changeme"  # Override in production
  persistence:
    enabled: true
    size: 10Gi
alertmanager:
  enabled: false
nodeExporter:
  enabled: true
kube-state-metrics:
  enabled: true
# Managed Kubernetes (e.g. AKS, EKS, GKE) does not expose control plane
# endpoints. Disable these to avoid scrape errors.
kubeApiServer:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false

Install:

helm install observability prometheus-community/kube-prometheus-stack \
  -n observability --create-namespace \
  -f observability-values.yaml

Verify the pods come up:

kubectl get pods -n observability

You should see Prometheus, Grafana, the operator, kube-state-metrics, and node-exporter pods running. Node-exporter runs as a DaemonSet; pending pods are normal if nodes are still scaling.

Configure Prometheus to scrape Vantage

The vantage-selfhosted chart can create the requisite ServiceMonitor objects to monitor Vantage:

helm upgrade --install abbyy-vantage \
  # ...
  --set observability.prometheus.serviceMonitor.create=true

See the chart documentation for more options. The observability key is top-level and must not be nested under vantage.

Linkerd mTLS

With Linkerd’s default permissive policy, Prometheus can normally scrape the Linkerd-proxied Vantage endpoints over plaintext. Install Prometheus in the observability namespace, enable the Vantage ServiceMonitor, and confirm every Vantage target is UP. If a strict Linkerd authorization policy blocks the scrape, use one of these approaches:

Enable Linkerd injection for the observability namespace so Prometheus participates in the mesh.
Configure observability.prometheus.serviceMonitor.scheme: https and a tlsConfig that references the client certificates made available to Prometheus by your Linkerd configuration.

Do not enable KEDA until Prometheus returns current values for the Vantage metrics used by the ScaledObject resources. See Autoscaling with KEDA for the inspection queries and readiness checks.

Istio mTLS

When Vantage is placed in an Istio mesh using STRICT mTLS (the default when ingress.enabled is true and ingress.type is istio), Prometheus and the ServiceMonitor must be configured to scrape endpoints over mTLS, because Prometheus uses pod IPs directly rather than the service name. Configure the Prometheus pod(s) with the following annotations:

annotations:
  proxy.istio.io/config: |
    proxyMetadata:
      OUTPUT_CERTS: /etc/istio-output-certs
  sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'

and the following volume and volumeMount (names and paths are customizable; the examples below use these):

volumes:
  - name: istio-certs
    emptyDir:
      medium: Memory
volumeMounts:
  - name: istio-certs
    mountPath: /etc/prom-certs/
    readOnly: true

With kube-prometheus-stack, this can be done with the following values:

prometheus:
  prometheusSpec:
    podMetadata:
      annotations:
        # force Istio to generate certificates into a local volume directory
        "proxy.istio.io/config": |
          proxyMetadata:
            OUTPUT_CERTS: /etc/istio-output-certs
        "sidecar.istio.io/userVolumeMount": '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'
    # shared volume where Istio will write certificates
    volumes:
      - name: istio-certs
        emptyDir: {}
    volumeMounts:
      - name: istio-certs
        mountPath: /etc/prom-certs

Alternatively, see the official Istio docs. Next, install the vantage-selfhosted chart so the ServiceMonitor(s) are configured to use these certificates:

# values.vantage-selfhosted.yaml
observability:
  prometheus:
    serviceMonitor:
      scheme: https
      tlsConfig:
        caFile: /etc/prom-certs/root-cert.pem
        certFile: /etc/prom-certs/cert-chain.pem
        keyFile: /etc/prom-certs/key.pem
        insecureSkipVerify: true  # Prometheus does not support Istio security naming, thus skip verifying target pod certificate

Verify metrics are flowing

Port-forward to the Prometheus UI:

kubectl port-forward -n observability svc/observability-kube-prometheus-prometheus 9090:9090

Check that Vantage targets are discovered and healthy:

curl -s http://localhost:9090/api/v1/targets \
  | jq '[.data.activeTargets[] | select(.scrapePool // "" | contains("vantage"))] | group_by(.health) | map({health: .[0].health, count: length})'

Query for a Vantage application metric:

curl -s "http://localhost:9090/api/v1/query?query=process_cpu_count" \
  | jq '.data.result | length'

A non-zero result confirms metrics are being scraped. You can also browse targets at http://localhost:9090/targets to see individual scrape status.

Access Grafana

kubectl port-forward -n observability svc/observability-grafana 3000:80

Open http://localhost:3000. Default credentials are admin / the password set in the values file. Prometheus is pre-configured as a data source.

Available metrics

Vantage applications expose process, HTTP client/server, and ABBYY platform metrics through the OpenTelemetry SDK, all labeled with service, pod, namespace, and otel_scope_name. See Available metrics for the full catalog.

OTLP ingestion

OTLP ingestion is push-based and independent of the scrape-based ServiceMonitor flow above. Use it only for services that push OpenTelemetry metrics rather than exposing a scrape endpoint.

When Prometheus is configured with the OTLP receiver enabled (--web.enable-otlp-receiver), any service that pushes OTLP metrics via HTTP can send them to:

http://observability-kube-prometheus-prometheus.observability:9090/api/v1/otlp/v1/metrics

​How Vantage exposes metrics

​Prerequisites

​Configure Prometheus

​Configure Prometheus to scrape Vantage

​Linkerd mTLS

​Istio mTLS

​Verify metrics are flowing

​Access Grafana

​Available metrics

​OTLP ingestion

How Vantage exposes metrics

Prerequisites

Configure Prometheus

Configure Prometheus to scrape Vantage

Linkerd mTLS

Istio mTLS

Verify metrics are flowing

Access Grafana

Available metrics

OTLP ingestion