> ## Documentation Index > Fetch the complete documentation index at: https://docs.abbyy.com/llms.txt > Use this file to discover all available pages before exploring further. # Troubleshooting > Diagnose operator, ArgoCD, secrets, database, ingress, mesh, and autoscaling issues for Vantage 3.0 self-hosted. This page collects the failures we see most often during install and day-2 operation, with the diagnostic commands and the fixes. Each entry is structured **Symptom → Diagnosis → Fix**. If you get stuck, capture the operator logs, the `Vantage` resource status, and any failing ArgoCD application status before contacting your ABBYY account team: ```bash theme={null} # Operator logs (locate the operator pod, then read its logs) kubectl -n $operator_namespace get pods kubectl -n $operator_namespace logs $operator_pod_name --tail=500 # Vantage resource (status conditions and degradationReasons) kubectl -n $install_namespace get vantages.vantage.abbyy.com $release_name -o yaml # ArgoCD applications managed by the operator argocd app list -l vantage.abbyy.com/managed-by=vantage-operator ``` ## Operator and CRD ### Force a retry after fixing an out-of-resource issue **Symptom.** You've fixed the underlying cause of a failure (added a missing Key Vault secret, granted RBAC, repaired a database) but the operator hasn't reconciled. **Diagnosis.** The operator responds to events on the `Vantage` resource itself; state changes outside the resource (Key Vault permissions, network policies, external services) do not trigger a reconcile. `Vantage` resources are mutable, but to retry installation of Vantage, the resource must be annotated with `vantage.abbyy.com/reprocess` (any value). **Fix.** Annotate the `Vantage` resource with `vantage.abbyy.com/reprocess` (any value) to force a fresh reconcile from the Preflight phase: ```bash theme={null} kubectl -n $install_namespace annotate vantages.vantage.abbyy.com/$release_name \ vantage.abbyy.com/reprocess="$(date +%s)" --overwrite ``` Update the annotation value (e.g., a timestamp) each time you want to retry. The operator only re-reconciles when the annotation value changes. ### `Vantage` resource stuck in `Progressing` **Symptom.** `kubectl describe vantages.vantage.abbyy.com $release_name -n $install_namespace` shows `Progressing=True` for an extended period and `Ready` is never `True`. **Diagnosis.** Inspect `status.degradationReasons[]` on the resource; the operator surfaces the failing component there. Then check the corresponding ArgoCD application: ```bash theme={null} kubectl -n $install_namespace get vantages.vantage.abbyy.com $release_name \ -o jsonpath='{.status.degradationReasons}' | jq . ``` **Fix.** Resolve the underlying ArgoCD sync error. Common causes: an upstream Helm chart needs a value the cluster does not yet have (Key Vault secret missing, [image pull failing](#pods-stuck-in-imagepullbackoff)), or an ingress gateway is not present. After the underlying issue is fixed, annotate the `Vantage` resource with `vantage.abbyy.com/reprocess` to retry the installation ([Force a retry](#force-a-retry-after-fixing-an-out-of-resource-issue)). ### Operator never picks up a new `Vantage` resource **Symptom.** You apply a `Vantage` resource and `kubectl get vantages` shows it, but the operator never reports any status. **Diagnosis.** Check that the operator is running and reconciling: ```bash theme={null} kubectl -n $operator_namespace get pods kubectl -n $operator_namespace logs $operator_pod_name | grep -i 'watching\|reconcile' ``` **Fix.** The operator watches `Vantage` resources cluster-wide; the chart exposes no value that restricts which namespaces it watches. If the operator is running but logs show no reconcile attempts for your resource, verify the CRD is at the version the operator expects (`vantage.abbyy.com/v1alpha1`) and that exactly one operator instance is installed in the cluster. To see the values the chart actually accepts, run `helm show values $charts_uri/vantage-operator --version $operator_version` ([Helm chart values](/vantage/self-hosted/v3.0/helm-chart#inspecting-available-values)). ### CRD apply fails with validation error **Symptom.** `kubectl apply` rejects the `Vantage` resource with a CEL validation message. **Diagnosis.** The error message comes from the CRD's CEL rules. Common causes are an `ociMigration` block with `enabled: true` but no `destination`, or a `techcore` block with `keepPreviousVersion: true` but no `previousVersion`. **Fix.** Set the field the validation message names. See [API reference](/vantage/self-hosted/v3.0/api-reference#vantagespec) for the full validation rules. ## OCI migration ### OCI migration job never starts **Symptom.** The `Vantage` resource stays `Progressing`; no job named `$release_name-oci-migration` exists in your install namespace. **Diagnosis.** Check the operator preflight checks in the operator logs and the resource's degradation reasons. The job is created only after preflight passes. **Fix.** Resolve the failing preflight check. During preflight checks, configuration is validated, including the secrets configuration, so start by confirming every required alias exists in your secrets provider ([Secrets and Key Vault](/vantage/self-hosted/v3.0/secrets-and-keyvault#secret-aliases)). ### OCI migration job runs but fails **Symptom.** `kubectl -n $install_namespace get jobs` shows the migration job, but it does not reach `Complete`. **Diagnosis.** Inspect the job logs: ```bash theme={null} kubectl -n $install_namespace logs job/$release_name-oci-migration --tail=200 ``` **Fix.** Most failures are auth-related: the job either could not pull from the source registry (check each `vantage.ociMigration.sources[]` entry's `credentialsRef.name` secret or username/password) or could not push to the destination (check `vantage.ociMigration.destination` credentials and the workload identity's RBAC if you are using Azure Workload Identity). If the job pod itself never pulls its image, see [Pods stuck in `ImagePullBackOff`](#pods-stuck-in-imagepullbackoff). ## Image pulls ### Pods stuck in `ImagePullBackOff` **Symptom.** Operator pods, the OCI migration job pod, or Vantage workload pods report `ImagePullBackOff` or `ErrImagePull`, and `kubectl describe pod` shows `unauthorized: authentication required` from your registry. **Diagnosis.** Find the failing pods and the registry error: ```bash theme={null} kubectl -n $operator_namespace get pods | grep -Ei 'imagepull|errimage' kubectl -n $install_namespace get pods | grep -Ei 'imagepull|errimage' kubectl -n $install_namespace describe pod $pod_name kubectl -n $install_namespace get pod $pod_name -o jsonpath='{.spec.serviceAccountName}{"\n"}' ``` **Fix.** Give the failing component pull access to your registry: * **Operator pods**: upgrade the `vantage-operator` release with `--set "manager.imagePullSecrets[0].name=my-pull-secret"`. * **OCI migration job**: run the job under a ServiceAccount that carries the pull secret (`vantage.ociMigration.serviceAccountName=my-service-account`), or reference the secret on the source entry (`vantage.ociMigration.sources[0].credentialsRef.name=my-pull-secret`). * **Vantage workloads**: add the pull secret to the `imagePullSecrets` of the ServiceAccount named by `vantage.serviceAccountName`. If the failing pod reports another ServiceAccount, including `default`, grant pull access to that ServiceAccount or configure the component to use the intended Vantage ServiceAccount. * **On AKS**: instead of pull secrets, grant `AcrPull` on the registry to the cluster's kubelet identity. See [Azure image pull access](/vantage/self-hosted/v3.0/providers/azure#image-pull-access) or [self-managed registry access](/vantage/self-hosted/v3.0/providers/self-managed#configure-registry-access) for the full setup. ## Secrets ### Pods fail to start with `MountVolume.SetUp failed for volume "secrets-store"` **Symptom.** Vantage pods fail to start; events on the pod show CSI driver mount failures. **Diagnosis.** The Secrets Store CSI driver could not fetch a secret from the key vault. Most common causes: ```bash theme={null} kubectl -n $install_namespace describe pod ``` * The identity selected by `vantage.secrets.azure.identityMode` does not have read access to the named secret or certificate. * A Key Vault object name does not match the alias Vantage expects, and no `objects` mapping was supplied. * The Azure provider for the CSI driver is not installed. **Fix.** Verify the configured identity has `get` permissions for every alias listed under [Secrets and Key Vault](/vantage/self-hosted/v3.0/secrets-and-keyvault#secret-aliases). If your Key Vault object names differ from the aliases, supply an `objects` mapping under `vantage.secrets.azure`. See [Identity modes](/vantage/self-hosted/v3.0/secrets-and-keyvault#identity-modes) for which identity is consulted in each mode. ### Paired TLS secrets are missing **Symptom.** Auth services fail to start, and one of the four TLS secret aliases is missing from your key vault. **Diagnosis.** Vantage requires four PEM-encoded materials stored as Key Vault secrets: a certificate and a private key for auth signing, and a certificate and a private key for auth deactivated tokens. **Fix.** Confirm all four aliases exist: `authSigningTlsCrt`, `authSigningTlsKey`, `authDeactivatedTlsCrt`, `authDeactivatedTlsKey`. See [Secrets and Key Vault → Certificate aliases](/vantage/self-hosted/v3.0/secrets-and-keyvault#certificate-aliases). ### Kubernetes Secret provider fails preflight **Symptom.** A self-managed install fails during preflight with a missing Secret, missing key, or `forbidden` error. **Diagnosis.** The operator reads native Secrets in `$install_namespace` during preflight. The named Secret may not exist, an `objects` mapping may point to the wrong name, the Secret may not contain the expected data key, or the chart-created RoleBinding may not identify the installed operator ServiceAccount. Check the operator's permission for one of the affected Secrets: ```bash theme={null} kubectl auth can-i get secret/$secret_name \ --as=system:serviceaccount:$operator_namespace:$operator_service_account \ --namespace $install_namespace ``` **Fix.** Confirm every Secret exists in the install namespace, then compare `vantage.secrets.kubernetes.objects` with the [secret alias inventory](/vantage/self-hosted/v3.0/secrets-and-keyvault#secret-aliases). If authorization fails, confirm that the Vantage chart's `operator.namespace` and `operator.controllerManagerServiceAccount` values identify the installed operator. If an external secrets controller creates the Secrets, wait for it to report a successful sync before forcing a new operator reconcile. ## Databases ### PostgreSQL workloads report `DataProvider` not found **Symptom.** Database startup or migration pods fail with `DataProvider '' not found`. **Diagnosis.** `vantage.databaseProvider` is case-sensitive. The PostgreSQL runtime provider is named `PostgreSQL` exactly. **Fix.** Set: ```yaml theme={null} vantage: databaseProvider: PostgreSQL reportingEnabled: false ``` Values such as `Postgresql` and `PostgreSql` are invalid at runtime. Apply the corrected values and [force a retry](#force-a-retry-after-fixing-an-out-of-resource-issue). ### PostgreSQL schema is not up to date **Symptom.** Several services fail startup with `postgres database schema is not up to date` or migration-version conflicts. **Diagnosis.** Multiple services are using the same logical PostgreSQL database. Each service maintains its own migration history and must use a distinct database. **Fix.** Create a separate database for every database alias. For workloads that consume the generic `Database__ConnectionString` key, create separate Kubernetes Secrets and map each alias with `vantage.secrets.kubernetes.objects`. See [PostgreSQL and per-service Secrets](/vantage/self-hosted/v3.0/secrets-and-keyvault#postgresql-and-per-service-secrets). ## ArgoCD ### ArgoCD application shows `OutOfSync` after install **Symptom.** `kubectl get application -A` reports one of the operator-managed applications as `OutOfSync` after the operator has otherwise reported `Ready`. **Diagnosis.** Look at the application's `status.conditions[]`: ```bash theme={null} argocd app get ``` **Fix.** If the drift is due to manual edits to operator-managed objects, revert them; the operator owns those objects and will re-sync. If the drift is in a chart value, update it through the `Vantage` resource (or the `vantage-selfhosted` Helm release). Never edit the underlying ArgoCD `Application` object directly. ### ArgoCD ApplicationSet sync stuck **Symptom.** ArgoCD applications never advance through their sync waves. **Diagnosis.** Verify ApplicationSet Progressive Syncs is enabled on your ArgoCD installation: ```bash theme={null} kubectl -n $argocd_namespace get configmap argocd-cmd-params-cm -o yaml | grep -i progressive ``` **Fix.** Enable Progressive Syncs per the [ArgoCD documentation](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Progressive-Syncs/#enabling-progressive-syncs) and restart the ApplicationSet controller. ## KEDA and Prometheus ### `ScaledObject` resources cannot create HPAs **Symptom.** KEDA reports `Ready=False` or `ScaledObjectCheckFailed`, and ArgoCD shows the affected workload applications as degraded. **Diagnosis.** Check the installed KEDA version and the `ScaledObject` events: ```bash theme={null} kubectl get deployment --all-namespaces | grep keda-operator kubectl -n $install_namespace describe scaledobject ``` The current Vantage workload charts require KEDA 2.17.x; KEDA 2.18 and later removed a scaler field that they use. **Fix.** Install KEDA 2.17.3, then re-sync the affected ArgoCD applications. ### HPA metrics show `` **Symptom.** KEDA resources exist, but `kubectl get hpa` shows `` for Vantage metrics and workloads do not scale as expected. **Diagnosis.** The `ScaledObject` resources query `http://prometheus-operated.observability.svc.cluster.local:9090`. The Service may be missing, the Vantage `ServiceMonitor` may not be selected, or the service mesh may block the scrape. **Fix.** Confirm that the Prometheus Service exists, its Vantage targets are `UP`, and the `observability` chart values are top-level: ```bash theme={null} kubectl -n observability get service prometheus-operated kubectl -n $install_namespace get servicemonitor ``` See [Autoscaling with KEDA](/vantage/self-hosted/v3.0/performance/autoscaling) for trigger behavior and verification, and [Monitoring with Prometheus](/vantage/self-hosted/v3.0/monitoring/prometheus) for Istio and Linkerd configuration. ## Skill installation ### Skills are missing after the install reports `Ready` **Symptom.** The `Vantage` resource reports `Ready` and Vantage is reachable, but expected skills are absent, and nothing skill-related appears in ArgoCD. **Diagnosis.** Skills are installed by a separate Kubernetes job that is not surfaced via ArgoCD, so it never appears in `argocd app list` (including the diagnostic capture at the top of this page). The job can still be running after `Ready`; you can use Vantage in the meantime. Check the job: ```bash theme={null} kubectl -n $install_namespace wait --for=create job/$install_namespace-skill-installer-job kubectl -n $install_namespace wait --for=condition=Complete job/$install_namespace-skill-installer-job kubectl -n $install_namespace logs job/$install_namespace-skill-installer-job --tail=200 ``` **Fix.** If the job is still running, wait for it to reach `Complete`. If the job failed, inspect its logs for the underlying error and resolve it; if the failure persists, capture the job logs alongside the operator logs for your ABBYY account team. ## Service mesh and ingress ### `503` from the ingress, application traffic blocked **Symptom.** `https://$your_vantage_hostname` returns `503` even though pods report `Ready`. **Diagnosis.** On AKS with the built-in Istio ingress, the external gateway may be unhealthy or the TLS Secret may not match `ingress.tls.secretName`. With customer-managed ingress, the controller may not have a matching route, the backend Service name or port may be wrong, or the TLS certificate may not be ready. **Fix.** For Istio, confirm the gateway and TLS Secret: ```bash theme={null} kubectl get gateway -A kubectl -n get pods | grep gateway ``` Confirm a `kubernetes.io/tls`-type Secret exists in the external gateway namespace with the name supplied to the chart, and that its certificate is valid for `dnsRecord`. For customer-managed ingress, inspect the controller logs and route status. Confirm that the route hostname equals `vantage.dnsRecord`, TLS is ready, and each backend Service exists in `$install_namespace`. See [Configure ingress and TLS](/vantage/self-hosted/v3.0/providers/self-managed#configure-ingress-and-tls). ### Sign-in redirects or OIDC metadata use `http://` **Symptom.** Vantage is served over HTTPS, but platform-administrator sign-in fails or `/auth2/.well-known/openid-configuration` advertises an `http://` issuer and endpoints. **Diagnosis.** TLS terminates at the ingress controller, but the authentication workloads are not honoring the forwarded HTTPS scheme. **Fix.** Confirm the ingress sends `X-Forwarded-Proto: https` and that the application containers for `auth-identity`, `auth-adminapi2`, `api-gateway`, and `api-registry` have: ```text theme={null} ASPNETCORE_FORWARDEDHEADERS_ENABLED=true ``` If your workload chart does not expose this environment setting, apply your approved admission mutation before restarting the affected pods. See [Forwarded HTTPS scheme](/vantage/self-hosted/v3.0/providers/self-managed#forwarded-https-scheme). ### Prometheus scrape targets show as `DOWN` for `/metrics-text` **Symptom.** In the in-mesh Prometheus, Vantage application targets show as `DOWN`. **Diagnosis.** On Istio, the Prometheus pod may be missing its sidecar or the Istio output certificates may not be written to `/etc/prom-certs/`. On Linkerd, a strict authorization policy may block plaintext scrapes from an unmeshed Prometheus. ```bash theme={null} kubectl -n observability get pods -o jsonpath='{.items[*].spec.containers[*].name}' ``` **Fix.** With Istio, verify the Prometheus pod has three containers (`prometheus`, `config-reloader`, `istio-proxy`) and that the `proxy.istio.io/config` annotation with `OUTPUT_CERTS: /etc/istio-output-certs` is present. With Linkerd, inject the `observability` namespace or configure the `ServiceMonitor` and Prometheus with the client certificates required by your policy. See [Monitoring with Prometheus](/vantage/self-hosted/v3.0/monitoring/prometheus). ## Day-2 operations ### After upgrade, the operator reports the new version but Vantage is still on the old one **Symptom.** `status.installedVantageVersion` is unchanged after upgrading the Helm releases. **Diagnosis.** The `vantage-operator` and `vantage-selfhosted` releases may not be on the same new version, or ArgoCD may still be applying the per-component changes. Check both Helm releases and the ArgoCD application sync status. **Fix.** Upgrade both charts to the same version, then wait for the ArgoCD applications to finish syncing. If the operator surfaces a `Degraded` condition, follow the degradation reason to the failing component, resolve it, then annotate the `Vantage` resource with `vantage.abbyy.com/reprocess` to retry ([Force a retry](#force-a-retry-after-fixing-an-out-of-resource-issue)). See [Upgrading](/vantage/self-hosted/v3.0/upgrading). ## What's next What the operator does on every reconcile, useful context when diagnosing. Current constraints that shape what is and is not actually a bug.