> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abbyy.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Azure Managed Monitoring

> Collect Vantage 3.0 self-hosted metrics on AKS using Azure Monitor Workspace, Azure Managed Grafana, and an in-mesh Prometheus that scrapes over Istio mTLS.

This guide walks you through collecting Vantage application metrics on an AKS cluster that already has Azure managed monitoring enabled (Azure Monitor Workspace + Managed Grafana + the `ama-metrics` agent). When complete, Vantage application metrics appear alongside your infrastructure metrics in the same Azure Managed Grafana, scraped by an in-mesh Prometheus that reaches Vantage's endpoints over Istio mTLS.

<Note>
  For background on why this pattern exists (including the Istio mTLS constraint that drives it), see [Monitoring Overview](/vantage/self-hosted/v3.0/monitoring/overview).
</Note>

## Why this pattern exists

The `ama-metrics` agent deployed by Azure handles Kubernetes infrastructure metrics (kubelet, cadvisor, kube-state-metrics) automatically. It can also scrape Istio proxy metrics like `istio_requests_total` via pod annotations, since those are exposed on cleartext ports by the Envoy sidecar.

Vantage application metrics are different: they're exposed on container port 8080, which is intercepted by the Envoy sidecar and subject to STRICT mTLS enforcement. The `ama-metrics` agent runs in `kube-system` without an Istio sidecar and cannot present valid mTLS certificates, so it cannot scrape application metrics endpoints in meshed namespaces. Microsoft documents this as a known limitation: ["Currently, there's no support to collect metrics from Istio setup with mutual TLS authentication."](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-istio-integration#limitations)

To collect Vantage application metrics, you deploy a Prometheus instance **inside the Istio mesh**. This Prometheus runs with an Istio sidecar, exports the mesh certificates, uses them to scrape application endpoints over mTLS, and remote-writes to the same Azure Monitor Workspace that `ama-metrics` uses.

## How it works

Vantage applications are instrumented with OpenTelemetry SDKs and expose Prometheus-format metrics on `/metrics-text` at container port 8080. Services that expose metrics carry the label `abbyy.platform.metrics.text.endpoint: "1"`.

The in-mesh Prometheus scrapes those metrics by:

1. Running with an Istio sidecar (`sidecar.istio.io/inject: "true"`).
2. Exporting Istio's mTLS certificates via the `OUTPUT_CERTS` proxy config annotation.
3. Mounting those certificates into the Prometheus container at `/etc/prom-certs/`.
4. Configuring the `ServiceMonitor` (created by the `vantage-selfhosted` chart in Step 6) to scrape over `scheme: https` using the Istio CA cert, client cert, and key.

## Architecture

| Component                               | Role                                                                                                                   |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Azure Monitor Workspace**             | Prometheus-compatible backend that stores all metrics.                                                                 |
| **Azure Managed Grafana**               | Visualization layer, pre-wired to the monitor workspace as a data source.                                              |
| **`ama-metrics` agent** (Azure-managed) | Scrapes default Kubernetes infrastructure metrics. Deployed automatically when Prometheus is enabled on AKS.           |
| **In-mesh Prometheus** (custom)         | Istio-injected Prometheus that scrapes Vantage app metrics over mTLS and remote-writes to the Azure Monitor Workspace. |
| **`ServiceMonitor`** (custom)           | Defines the Vantage scrape targets: `/metrics-text` on services labeled `abbyy.platform.metrics.text.endpoint: "1"`.   |

## Prerequisites

* AKS cluster with the Istio service mesh add-on enabled (STRICT mTLS).
* Azure Monitor Workspace provisioned and linked to the AKS cluster.
* Azure Managed Grafana provisioned and linked to the Azure Monitor Workspace.
* `ama-metrics` agent running on the cluster (enabled via AKS Monitor Settings).
* OIDC issuer enabled on the AKS cluster (required for workload identity).
* Prometheus Operator CRDs installed (these come with `ama-metrics`).

## Step 1: Provision Azure resources

If you haven't already created the monitoring resources, do so now in the Azure portal:

1. **Azure Monitor Workspace**: Portal → "Azure Monitor workspaces" → Create. Use the same resource group and region as your AKS cluster. Naming convention: `amw-<cluster-name>`.
2. **Azure Managed Grafana**: Portal → "Azure Managed Grafana" → Create. Same resource group and region. Link to the monitor workspace during creation. Naming convention: `amg-<cluster-name>`.
3. **Enable Prometheus on AKS**: Portal → AKS cluster → Monitoring → Monitor Settings → enable "Metrics via managed Prometheus" and select the monitor workspace and Grafana instance.

Verify that `ama-metrics` pods are running:

```bash theme={null}
kubectl get pods -n kube-system | grep ama-metrics
```

## Step 2: Create the observability namespace with Istio injection

> **Note:** Any namespace name can be used, but certain optional Vantage features (like KEDA support) depend on Prometheus services living in the `observability` namespace, so using `observability` is recommended.

```bash theme={null}
kubectl create namespace observability
```

Check the Istio revision label used by other meshed namespaces:

```bash theme={null}
# replace with your namespace if necesssary
kubectl get namespace app -o jsonpath='{.metadata.labels}' | grep istio
```

Apply the same label to the `observability` namespace:

```bash theme={null}
kubectl label namespace observability istio.io/rev=<revision>
```

## Step 3: Gather Azure Monitor Workspace ingestion details

Get the data collection rule and endpoint resource IDs:

```bash theme={null}
az monitor account show \
  --name <monitor-workspace-name> \
  --resource-group <resource-group> \
  --query "defaultIngestionSettings"
```

Get the metrics ingestion endpoint URL:

```bash theme={null}
az monitor data-collection endpoint show \
  --name <monitor-workspace-name> \
  --resource-group <managed-resource-group> \
  --query "metricsIngestion.endpoint"
```

Get the DCR immutable ID:

```bash theme={null}
az monitor data-collection rule show \
  --name <monitor-workspace-name> \
  --resource-group <managed-resource-group> \
  --query "immutableId"
```

<Note>
  The managed resource group follows the pattern `MA_<monitor-workspace-name>_<region>_managed`.
</Note>

Construct the full remote-write URL (you'll use it in Step 5):

```text theme={null}
https://<endpoint>/dataCollectionRules/<dcr-immutable-id>/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24
```

## Step 4: Create a managed identity for Prometheus

Create a user-assigned managed identity:

```bash theme={null}
az identity create \
  --name <cluster-name>-prometheus \
  --resource-group <resource-group> \
  --location <region>
```

Note the `clientId` and `principalId` from the output.

Assign the `Monitoring Metrics Publisher` role on the data collection rule:

```bash theme={null}
az role assignment create \
  --role "Monitoring Metrics Publisher" \
  --assignee-object-id <principalId> \
  --assignee-principal-type ServicePrincipal \
  --scope "<dataCollectionRuleResourceId>"
```

Get the OIDC issuer URL:

```bash theme={null}
az aks show \
  --name <cluster-name> \
  --resource-group <resource-group> \
  --query "oidcIssuerProfile.issuerUrl" -o tsv
```

Create a federated credential that ties the managed identity to the Prometheus service account:

```bash theme={null}
az identity federated-credential create \
  --name prometheus-federated \
  --identity-name <cluster-name>-prometheus \
  --resource-group <resource-group> \
  --issuer "<oidc-issuer-url>" \
  --subject "system:serviceaccount:observability:prometheus" \
  --audiences "api://AzureADTokenExchange"
```

## Step 5: Install the Prometheus Operator

The `ama-metrics` agent installs the Prometheus CRDs but does not run an operator that reconciles custom Prometheus resources. Install the operator only; disable everything else:

```bash theme={null}
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus-operator prometheus-community/kube-prometheus-stack \
  --namespace observability \
  --set prometheus.enabled=false \
  --set alertmanager.enabled=false \
  --set grafana.enabled=false \
  --set defaultRules.create=false \
  --set nodeExporter.enabled=false \
  --set kubeStateMetrics.enabled=false
```

## Step 6: Apply the Prometheus manifests

Three manifests are required (service account, RBAC, and the Prometheus instance). Apply them in the order shown. The Vantage `ServiceMonitor` is created separately by the `vantage-selfhosted` chart (see below).

**`service-account.yaml`**: Prometheus service account with workload identity annotation:

```yaml theme={null}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: observability
  annotations:
    azure.workload.identity/client-id: "<managed-identity-clientId>"
  labels:
    azure.workload.identity/use: "true"
```

**`rbac.yaml`**: `ClusterRole` and binding for target discovery:

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources: [nodes, nodes/metrics, services, endpoints, pods]
    verbs: [get, list, watch]
  - apiGroups: ["networking.k8s.io"]
    resources: [ingresses]
    verbs: [get, list, watch]
  - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: observability
```

**`prometheus.yaml`**: Prometheus instance with Istio sidecar injection, certificate export, and remote-write to Azure Monitor:

```yaml theme={null}
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: metrics
  namespace: observability
spec:
  replicas: 1
  scrapeInterval: 60s
  logLevel: debug
  serviceAccountName: prometheus
  podMetadata:
    annotations:
      proxy.istio.io/config: |
        proxyMetadata:
          OUTPUT_CERTS: /etc/istio-output-certs
      sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]'
    labels:
      sidecar.istio.io/inject: "true"
      azure.workload.identity/use: "true"
  volumes:
    - name: istio-certs
      emptyDir:
        medium: Memory
  volumeMounts:
    - name: istio-certs
      mountPath: /etc/prom-certs/
      readOnly: true
  resources:
    requests:
      memory: 3Gi
      cpu: 600m
    limits:
      memory: 6Gi
      cpu: 1000m
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  remoteWrite:
    - url: "<remote-write-url>"
      azureAd:
        cloud: AzurePublic
        sdk:
          tenantId: "<azure-tenant-id>"
  replicaExternalLabelName: "__replica__"
  externalLabels:
    cluster: "<cluster-name>"
```

<Note>
  The `azureAd.sdk` auth method uses `DefaultAzureCredential`, which picks up the workload identity token injected by the `azure.workload.identity/use: "true"` label. Requires Prometheus v3.60+.
</Note>

Apply all three:

```bash theme={null}
kubectl apply -f service-account.yaml
kubectl apply -f rbac.yaml
kubectl apply -f prometheus.yaml
```

### Configure the Vantage ServiceMonitor

The Vantage `ServiceMonitor` is created by the `vantage-selfhosted` Helm chart rather than applied by hand. Install or upgrade the chart with a values file that configures the `ServiceMonitor` to scrape Vantage over Istio mTLS using the certificates the in-mesh Prometheus exports:

```yaml theme={null}
# values.vantage-selfhosted.yaml
observability:
  prometheus:
    serviceMonitor:
      scheme: https
      tlsConfig:
        caFile: /etc/prom-certs/root-cert.pem
        certFile: /etc/prom-certs/cert-chain.pem
        keyFile: /etc/prom-certs/key.pem
        insecureSkipVerify: true  # Prometheus does not support Istio security naming, thus skip verifying target pod certificate
      relabelings:
        - action: replace
          sourceLabels: [__meta_kubernetes_service_name]
          targetLabel: service_name
      metricRelabelings:
        - action: replace
          sourceLabels: [__meta_kubernetes_service_name]
          targetLabel: app_service
```

For the full explanation of the Istio mTLS certificate wiring these paths rely on, see [Monitoring with Prometheus](/vantage/self-hosted/v3.0/monitoring/prometheus#istio-mtls).

## Step 7: Verify

Check that the Prometheus pod is running with three containers (`prometheus`, `config-reloader`, `istio-proxy`):

```bash theme={null}
kubectl get pods -n observability
```

Port-forward to inspect scrape targets:

```bash theme={null}
kubectl port-forward -n observability prometheus-metrics-0 9090:9090
```

Open `http://localhost:9090/targets`. Targets should show as **UP**, scraping over `https` on `/metrics-text:8080`.

Check remote-write logs for errors:

```bash theme={null}
kubectl logs -n observability prometheus-metrics-0 -c prometheus | grep -i "remote" | tail -10
```

Successful remote-write shows "Done replaying WAL" with no auth errors.

In Managed Grafana, query `up{cluster="<cluster-name>"}` in the Explore view using the Managed Prometheus data source to confirm metrics are flowing.

## Operational notes

* The `ama-metrics` agent continues handling Kubernetes infrastructure metrics (kubelet, cadvisor, kube-state-metrics, node-exporter). Do **not** disable it.
* The in-mesh Prometheus only handles Vantage application metrics. Both write to the same Azure Monitor Workspace.
* The `azureAd.sdk` remote-write auth uses `DefaultAzureCredential`, which picks up the workload identity token projected into the pod by the AKS workload identity webhook. This requires the service account annotation `azure.workload.identity/client-id` and the pod label `azure.workload.identity/use: "true"`.
* The `azureAd.managedIdentity` config does **not** work for this use case: it calls the IMDS endpoint on the node, and the user-assigned identity is not assigned to the VMSS node pool.
* `UNKNOWN` targets in the Prometheus targets view are typically pods that are in a Degraded or CrashLooping state, not a scraping issue.
* `logLevel: debug` in the Prometheus CRD can be changed to `info` once the setup is verified.