Azure Managed Monitoring - ABBYY Documentation

This guide walks you through collecting Vantage application metrics on an AKS cluster that already has Azure managed monitoring enabled (Azure Monitor Workspace + Managed Grafana + the ama-metrics agent). When complete, Vantage application metrics appear alongside your infrastructure metrics in the same Azure Managed Grafana, scraped by an in-mesh Prometheus that reaches Vantage’s endpoints over Istio mTLS.

For background on why this pattern exists (including the Istio mTLS constraint that drives it), see Monitoring Overview.

Why this pattern exists

The ama-metrics agent deployed by Azure handles Kubernetes infrastructure metrics (kubelet, cadvisor, kube-state-metrics) automatically. It can also scrape Istio proxy metrics like istio_requests_total via pod annotations, since those are exposed on cleartext ports by the Envoy sidecar. Vantage application metrics are different: they’re exposed on container port 8080, which is intercepted by the Envoy sidecar and subject to STRICT mTLS enforcement. The ama-metrics agent runs in kube-system without an Istio sidecar and cannot present valid mTLS certificates, so it cannot scrape application metrics endpoints in meshed namespaces. Microsoft documents this as a known limitation: “Currently, there’s no support to collect metrics from Istio setup with mutual TLS authentication.” To collect Vantage application metrics, you deploy a Prometheus instance inside the Istio mesh. This Prometheus runs with an Istio sidecar, exports the mesh certificates, uses them to scrape application endpoints over mTLS, and remote-writes to the same Azure Monitor Workspace that ama-metrics uses.

How it works

Vantage applications are instrumented with OpenTelemetry SDKs and expose Prometheus-format metrics on /metrics-text at container port 8080. Services that expose metrics carry the label abbyy.platform.metrics.text.endpoint: "1". The in-mesh Prometheus scrapes those metrics by:

Running with an Istio sidecar (sidecar.istio.io/inject: "true").
Exporting Istio’s mTLS certificates via the OUTPUT_CERTS proxy config annotation.
Mounting those certificates into the Prometheus container at /etc/prom-certs/.
Configuring the ServiceMonitor (created by the vantage-selfhosted chart in Step 6) to scrape over scheme: https using the Istio CA cert, client cert, and key.

Architecture

Component	Role
Azure Monitor Workspace	Prometheus-compatible backend that stores all metrics.
Azure Managed Grafana	Visualization layer, pre-wired to the monitor workspace as a data source.
`ama-metrics` agent (Azure-managed)	Scrapes default Kubernetes infrastructure metrics. Deployed automatically when Prometheus is enabled on AKS.
In-mesh Prometheus (custom)	Istio-injected Prometheus that scrapes Vantage app metrics over mTLS and remote-writes to the Azure Monitor Workspace.
`ServiceMonitor` (custom)	Defines the Vantage scrape targets: `/metrics-text` on services labeled `abbyy.platform.metrics.text.endpoint: "1"`.

Prerequisites

AKS cluster with the Istio service mesh add-on enabled (STRICT mTLS).
Azure Monitor Workspace provisioned and linked to the AKS cluster.
Azure Managed Grafana provisioned and linked to the Azure Monitor Workspace.
ama-metrics agent running on the cluster (enabled via AKS Monitor Settings).
OIDC issuer enabled on the AKS cluster (required for workload identity).
Prometheus Operator CRDs installed (these come with ama-metrics).

Step 1: Provision Azure resources

If you haven’t already created the monitoring resources, do so now in the Azure portal:

Azure Monitor Workspace: Portal → “Azure Monitor workspaces” → Create. Use the same resource group and region as your AKS cluster. Naming convention: amw-<cluster-name>.
Azure Managed Grafana: Portal → “Azure Managed Grafana” → Create. Same resource group and region. Link to the monitor workspace during creation. Naming convention: amg-<cluster-name>.
Enable Prometheus on AKS: Portal → AKS cluster → Monitoring → Monitor Settings → enable “Metrics via managed Prometheus” and select the monitor workspace and Grafana instance.

Verify that ama-metrics pods are running:

kubectl get pods -n kube-system | grep ama-metrics

Step 2: Create the observability namespace with Istio injection

Note: Any namespace name can be used, but certain optional Vantage features (like KEDA support) depend on Prometheus services living in the observability namespace, so using observability is recommended.

kubectl create namespace observability

Check the Istio revision label used by other meshed namespaces:

# replace with your namespace if necesssary
kubectl get namespace app -o jsonpath='{.metadata.labels}' | grep istio

Apply the same label to the observability namespace:

kubectl label namespace observability istio.io/rev=<revision>

Step 3: Gather Azure Monitor Workspace ingestion details

Get the data collection rule and endpoint resource IDs:

az monitor account show \
  --name <monitor-workspace-name> \
  --resource-group <resource-group> \
  --query "defaultIngestionSettings"

Get the metrics ingestion endpoint URL:

az monitor data-collection endpoint show \
  --name <monitor-workspace-name> \
  --resource-group <managed-resource-group> \
  --query "metricsIngestion.endpoint"

Get the DCR immutable ID:

az monitor data-collection rule show \
  --name <monitor-workspace-name> \
  --resource-group <managed-resource-group> \
  --query "immutableId"

The managed resource group follows the pattern MA_<monitor-workspace-name>_<region>_managed.

Construct the full remote-write URL (you’ll use it in Step 5):

https://<endpoint>/dataCollectionRules/<dcr-immutable-id>/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24

Step 4: Create a managed identity for Prometheus

Create a user-assigned managed identity:

az identity create \
  --name <cluster-name>-prometheus \
  --resource-group <resource-group> \
  --location <region>

Note the clientId and principalId from the output. Assign the Monitoring Metrics Publisher role on the data collection rule:

az role assignment create \
  --role "Monitoring Metrics Publisher" \
  --assignee-object-id <principalId> \
  --assignee-principal-type ServicePrincipal \
  --scope "<dataCollectionRuleResourceId>"

Get the OIDC issuer URL:

az aks show \
  --name <cluster-name> \
  --resource-group <resource-group> \
  --query "oidcIssuerProfile.issuerUrl" -o tsv

Create a federated credential that ties the managed identity to the Prometheus service account:

az identity federated-credential create \
  --name prometheus-federated \
  --identity-name <cluster-name>-prometheus \
  --resource-group <resource-group> \
  --issuer "<oidc-issuer-url>" \
  --subject "system:serviceaccount:observability:prometheus" \
  --audiences "api://AzureADTokenExchange"

Step 5: Install the Prometheus Operator

The ama-metrics agent installs the Prometheus CRDs but does not run an operator that reconciles custom Prometheus resources. Install the operator only; disable everything else:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus-operator prometheus-community/kube-prometheus-stack \
  --namespace observability \
  --set prometheus.enabled=false \
  --set alertmanager.enabled=false \
  --set grafana.enabled=false \
  --set defaultRules.create=false \
  --set nodeExporter.enabled=false \
  --set kubeStateMetrics.enabled=false

Step 6: Apply the Prometheus manifests

Three manifests are required (service account, RBAC, and the Prometheus instance). Apply them in the order shown. The Vantage ServiceMonitor is created separately by the vantage-selfhosted chart (see below). service-account.yaml: Prometheus service account with workload identity annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: observability
  annotations:
    azure.workload.identity/client-id: "<managed-identity-clientId>"
  labels:
    azure.workload.identity/use: "true"

rbac.yaml: ClusterRole and binding for target discovery:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources: [nodes, nodes/metrics, services, endpoints, pods]
    verbs: [get, list, watch]
  - apiGroups: ["networking.k8s.io"]
    resources: [ingresses]
    verbs: [get, list, watch]
  - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: observability

prometheus.yaml: Prometheus instance with Istio sidecar injection, certificate export, and remote-write to Azure Monitor:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: metrics
  namespace: observability
spec:
  replicas: 1
  scrapeInterval: 60s
  logLevel: debug
  serviceAccountName: prometheus
  podMetadata:
    annotations:
      proxy.istio.io/config: |
        proxyMetadata:
          OUTPUT_CERTS: /etc/istio-output-certs
      sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'
    labels:
      sidecar.istio.io/inject: "true"
      azure.workload.identity/use: "true"
  volumes:
    - name: istio-certs
      emptyDir:
        medium: Memory
  volumeMounts:
    - name: istio-certs
      mountPath: /etc/prom-certs/
      readOnly: true
  resources:
    requests:
      memory: 3Gi
      cpu: 600m
    limits:
      memory: 6Gi
      cpu: 1000m
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  remoteWrite:
    - url: "<remote-write-url>"
      azureAd:
        cloud: AzurePublic
        sdk:
          tenantId: "<azure-tenant-id>"
  replicaExternalLabelName: "__replica__"
  externalLabels:
    cluster: "<cluster-name>"

The azureAd.sdk auth method uses DefaultAzureCredential, which picks up the workload identity token injected by the azure.workload.identity/use: "true" label. Requires Prometheus v3.60+.

Apply all three:

kubectl apply -f service-account.yaml
kubectl apply -f rbac.yaml
kubectl apply -f prometheus.yaml

Configure the Vantage ServiceMonitor

The Vantage ServiceMonitor is created by the vantage-selfhosted Helm chart rather than applied by hand. Install or upgrade the chart with a values file that configures the ServiceMonitor to scrape Vantage over Istio mTLS using the certificates the in-mesh Prometheus exports:

# values.vantage-selfhosted.yaml
observability:
  prometheus:
    serviceMonitor:
      scheme: https
      tlsConfig:
        caFile: /etc/prom-certs/root-cert.pem
        certFile: /etc/prom-certs/cert-chain.pem
        keyFile: /etc/prom-certs/key.pem
        insecureSkipVerify: true  # Prometheus does not support Istio security naming, thus skip verifying target pod certificate
      relabelings:
        - action: replace
          sourceLabels: [__meta_kubernetes_service_name]
          targetLabel: service_name
      metricRelabelings:
        - action: replace
          sourceLabels: [__meta_kubernetes_service_name]
          targetLabel: app_service

For the full explanation of the Istio mTLS certificate wiring these paths rely on, see Monitoring with Prometheus.

Step 7: Verify

Check that the Prometheus pod is running with three containers (prometheus, config-reloader, istio-proxy):

kubectl get pods -n observability

Port-forward to inspect scrape targets:

kubectl port-forward -n observability prometheus-metrics-0 9090:9090

Open http://localhost:9090/targets. Targets should show as UP, scraping over https on /metrics-text:8080. Check remote-write logs for errors:

kubectl logs -n observability prometheus-metrics-0 -c prometheus | grep -i "remote" | tail -10

Successful remote-write shows “Done replaying WAL” with no auth errors. In Managed Grafana, query up{cluster="<cluster-name>"} in the Explore view using the Managed Prometheus data source to confirm metrics are flowing.

Operational notes

The ama-metrics agent continues handling Kubernetes infrastructure metrics (kubelet, cadvisor, kube-state-metrics, node-exporter). Do not disable it.
The in-mesh Prometheus only handles Vantage application metrics. Both write to the same Azure Monitor Workspace.
The azureAd.sdk remote-write auth uses DefaultAzureCredential, which picks up the workload identity token projected into the pod by the AKS workload identity webhook. This requires the service account annotation azure.workload.identity/client-id and the pod label azure.workload.identity/use: "true".
The azureAd.managedIdentity config does not work for this use case: it calls the IMDS endpoint on the node, and the user-assigned identity is not assigned to the VMSS node pool.
UNKNOWN targets in the Prometheus targets view are typically pods that are in a Degraded or CrashLooping state, not a scraping issue.
logLevel: debug in the Prometheus CRD can be changed to info once the setup is verified.

​Why this pattern exists

​How it works

​Architecture

​Prerequisites

​Step 1: Provision Azure resources

​Step 2: Create the observability namespace with Istio injection

​Step 3: Gather Azure Monitor Workspace ingestion details

​Step 4: Create a managed identity for Prometheus

​Step 5: Install the Prometheus Operator

​Step 6: Apply the Prometheus manifests

​Configure the Vantage ServiceMonitor

​Step 7: Verify

​Operational notes

Why this pattern exists

How it works

Architecture

Prerequisites

Step 1: Provision Azure resources

Step 2: Create the observability namespace with Istio injection

Step 3: Gather Azure Monitor Workspace ingestion details

Step 4: Create a managed identity for Prometheus

Step 5: Install the Prometheus Operator

Step 6: Apply the Prometheus manifests

Configure the Vantage ServiceMonitor

Step 7: Verify

Operational notes