Skip to main content
This page collects the failures we see most often during install and day-2 operation, with the diagnostic commands and the fixes. Each entry is structured Symptom → Diagnosis → Fix. If you get stuck, capture the operator logs, the Vantage resource status, and any failing ArgoCD application status before contacting your ABBYY account team:

​
Operator and CRD

​
Force a retry after fixing an out-of-resource issue

Symptom. You’ve fixed the underlying cause of a failure (added a missing Key Vault secret, granted RBAC, repaired a database) but the operator hasn’t reconciled. Diagnosis. The operator responds to events on the Vantage resource itself; state changes outside the resource (Key Vault permissions, network policies, external services) do not trigger a reconcile. Vantage resources are mutable, but to retry installation of Vantage, the resource must be annotated with vantage.abbyy.com/reprocess (any value). Fix. Annotate the Vantage resource with vantage.abbyy.com/reprocess (any value) to force a fresh reconcile from the Preflight phase:
Update the annotation value (e.g., a timestamp) each time you want to retry. The operator only re-reconciles when the annotation value changes.

​
Vantage resource stuck in Progressing

Symptom. kubectl describe vantages.vantage.abbyy.com $release_name -n $install_namespace shows Progressing=True for an extended period and Ready is never True. Diagnosis. Inspect status.degradationReasons[] on the resource; the operator surfaces the failing component there. Then check the corresponding ArgoCD application:
Fix. Resolve the underlying ArgoCD sync error. Common causes: an upstream Helm chart needs a value the cluster does not yet have (Key Vault secret missing, image pull failing), or an ingress gateway is not present. After the underlying issue is fixed, annotate the Vantage resource with vantage.abbyy.com/reprocess to retry the installation (Force a retry).

​
Operator never picks up a new Vantage resource

Symptom. You apply a Vantage resource and kubectl get vantages shows it, but the operator never reports any status. Diagnosis. Check that the operator is running and reconciling:
Fix. The operator watches Vantage resources cluster-wide; the chart exposes no value that restricts which namespaces it watches. If the operator is running but logs show no reconcile attempts for your resource, verify the CRD is at the version the operator expects (vantage.abbyy.com/v1alpha1) and that exactly one operator instance is installed in the cluster. To see the values the chart actually accepts, run helm show values $charts_uri/vantage-operator --version $operator_version (Helm chart values).

​
CRD apply fails with validation error

Symptom. kubectl apply rejects the Vantage resource with a CEL validation message. Diagnosis. The error message comes from the CRD’s CEL rules. Common causes are an ociMigration block with enabled: true but no destination, or a techcore block with keepPreviousVersion: true but no previousVersion. Fix. Set the field the validation message names. See API reference for the full validation rules.

​
OCI migration

​
OCI migration job never starts

Symptom. The Vantage resource stays Progressing; no job named $release_name-oci-migration exists in your install namespace. Diagnosis. Check the operator preflight checks in the operator logs and the resource’s degradation reasons. The job is created only after preflight passes. Fix. Resolve the failing preflight check. During preflight checks, configuration is validated, including the secrets configuration, so start by confirming every required alias exists in your secrets provider (Secrets and Key Vault).

​
OCI migration job runs but fails

Symptom. kubectl -n $install_namespace get jobs shows the migration job, but it does not reach Complete. Diagnosis. Inspect the job logs:
Fix. Most failures are auth-related: the job either could not pull from the source registry (check each vantage.ociMigration.sources[] entry’s credentialsRef.name secret or username/password) or could not push to the destination (check vantage.ociMigration.destination credentials and the workload identity’s RBAC if you are using Azure Workload Identity). If the job pod itself never pulls its image, see Pods stuck in ImagePullBackOff.

​
Image pulls

​
Pods stuck in ImagePullBackOff

Symptom. Operator pods, the OCI migration job pod, or Vantage workload pods report ImagePullBackOff or ErrImagePull, and kubectl describe pod shows unauthorized: authentication required from your registry. Diagnosis. Find the failing pods and the registry error:
Fix. Give the failing component pull access to your registry:
  • Operator pods: upgrade the vantage-operator release with --set "manager.imagePullSecrets[0].name=my-pull-secret".
  • OCI migration job: run the job under a ServiceAccount that carries the pull secret (vantage.ociMigration.serviceAccountName=my-service-account), or reference the secret on the source entry (vantage.ociMigration.sources[0].credentialsRef.name=my-pull-secret).
  • Vantage workloads: add the pull secret to the imagePullSecrets of the ServiceAccount named by vantage.serviceAccountName. If the failing pod reports another ServiceAccount, including default, grant pull access to that ServiceAccount or configure the component to use the intended Vantage ServiceAccount.
  • On AKS: instead of pull secrets, grant AcrPull on the registry to the cluster’s kubelet identity.
See Azure image pull access or self-managed registry access for the full setup.

​
Secrets

​
Pods fail to start with MountVolume.SetUp failed for volume "secrets-store"

Symptom. Vantage pods fail to start; events on the pod show CSI driver mount failures. Diagnosis. The Secrets Store CSI driver could not fetch a secret from the key vault. Most common causes:
  • The identity selected by vantage.secrets.azure.identityMode does not have read access to the named secret or certificate.
  • A Key Vault object name does not match the alias Vantage expects, and no objects mapping was supplied.
  • The Azure provider for the CSI driver is not installed.
Fix. Verify the configured identity has get permissions for every alias listed under Secrets and Key Vault. If your Key Vault object names differ from the aliases, supply an objects mapping under vantage.secrets.azure. See Identity modes for which identity is consulted in each mode.

​
Paired TLS secrets are missing

Symptom. Auth services fail to start, and one of the four TLS secret aliases is missing from your key vault. Diagnosis. Vantage requires four PEM-encoded materials stored as Key Vault secrets: a certificate and a private key for auth signing, and a certificate and a private key for auth deactivated tokens. Fix. Confirm all four aliases exist: authSigningTlsCrt, authSigningTlsKey, authDeactivatedTlsCrt, authDeactivatedTlsKey. See Secrets and Key Vault → Certificate aliases.

​
Kubernetes Secret provider fails preflight

Symptom. A self-managed install fails during preflight with a missing Secret, missing key, or forbidden error. Diagnosis. The operator reads native Secrets in $install_namespace during preflight. The named Secret may not exist, an objects mapping may point to the wrong name, the Secret may not contain the expected data key, or the chart-created RoleBinding may not identify the installed operator ServiceAccount. Check the operator’s permission for one of the affected Secrets:
Fix. Confirm every Secret exists in the install namespace, then compare vantage.secrets.kubernetes.objects with the secret alias inventory. If authorization fails, confirm that the Vantage chart’s operator.namespace and operator.controllerManagerServiceAccount values identify the installed operator. If an external secrets controller creates the Secrets, wait for it to report a successful sync before forcing a new operator reconcile.

​
Databases

​
PostgreSQL workloads report DataProvider not found

Symptom. Database startup or migration pods fail with DataProvider '<value>' not found. Diagnosis. vantage.databaseProvider is case-sensitive. The PostgreSQL runtime provider is named PostgreSQL exactly. Fix. Set:
Values such as Postgresql and PostgreSql are invalid at runtime. Apply the corrected values and force a retry.

​
PostgreSQL schema is not up to date

Symptom. Several services fail startup with postgres database schema is not up to date or migration-version conflicts. Diagnosis. Multiple services are using the same logical PostgreSQL database. Each service maintains its own migration history and must use a distinct database. Fix. Create a separate database for every database alias. For workloads that consume the generic Database__ConnectionString key, create separate Kubernetes Secrets and map each alias with vantage.secrets.kubernetes.objects. See PostgreSQL and per-service Secrets.

​
ArgoCD

​
ArgoCD application shows OutOfSync after install

Symptom. kubectl get application -A reports one of the operator-managed applications as OutOfSync after the operator has otherwise reported Ready. Diagnosis. Look at the application’s status.conditions[]:
Fix. If the drift is due to manual edits to operator-managed objects, revert them; the operator owns those objects and will re-sync. If the drift is in a chart value, update it through the Vantage resource (or the vantage-selfhosted Helm release). Never edit the underlying ArgoCD Application object directly.

​
ArgoCD ApplicationSet sync stuck

Symptom. ArgoCD applications never advance through their sync waves. Diagnosis. Verify ApplicationSet Progressive Syncs is enabled on your ArgoCD installation:
Fix. Enable Progressive Syncs per the ArgoCD documentation and restart the ApplicationSet controller.

​
KEDA and Prometheus

​
ScaledObject resources cannot create HPAs

Symptom. KEDA reports Ready=False or ScaledObjectCheckFailed, and ArgoCD shows the affected workload applications as degraded. Diagnosis. Check the installed KEDA version and the ScaledObject events:
The current Vantage workload charts require KEDA 2.17.x; KEDA 2.18 and later removed a scaler field that they use. Fix. Install KEDA 2.17.3, then re-sync the affected ArgoCD applications.

​
HPA metrics show <unknown>

Symptom. KEDA resources exist, but kubectl get hpa shows <unknown> for Vantage metrics and workloads do not scale as expected. Diagnosis. The ScaledObject resources query http://prometheus-operated.observability.svc.cluster.local:9090. The Service may be missing, the Vantage ServiceMonitor may not be selected, or the service mesh may block the scrape. Fix. Confirm that the Prometheus Service exists, its Vantage targets are UP, and the observability chart values are top-level:
See Autoscaling with KEDA for trigger behavior and verification, and Monitoring with Prometheus for Istio and Linkerd configuration.

​
Skill installation

​
Skills are missing after the install reports Ready

Symptom. The Vantage resource reports Ready and Vantage is reachable, but expected skills are absent, and nothing skill-related appears in ArgoCD. Diagnosis. Skills are installed by a separate Kubernetes job that is not surfaced via ArgoCD, so it never appears in argocd app list (including the diagnostic capture at the top of this page). The job can still be running after Ready; you can use Vantage in the meantime. Check the job:
Fix. If the job is still running, wait for it to reach Complete. If the job failed, inspect its logs for the underlying error and resolve it; if the failure persists, capture the job logs alongside the operator logs for your ABBYY account team.

​
Service mesh and ingress

​
503 from the ingress, application traffic blocked

Symptom. https://$your_vantage_hostname returns 503 even though pods report Ready. Diagnosis. On AKS with the built-in Istio ingress, the external gateway may be unhealthy or the TLS Secret may not match ingress.tls.secretName. With customer-managed ingress, the controller may not have a matching route, the backend Service name or port may be wrong, or the TLS certificate may not be ready. Fix. For Istio, confirm the gateway and TLS Secret:
Confirm a kubernetes.io/tls-type Secret exists in the external gateway namespace with the name supplied to the chart, and that its certificate is valid for dnsRecord. For customer-managed ingress, inspect the controller logs and route status. Confirm that the route hostname equals vantage.dnsRecord, TLS is ready, and each backend Service exists in $install_namespace. See Configure ingress and TLS.

​
Sign-in redirects or OIDC metadata use http://

Symptom. Vantage is served over HTTPS, but platform-administrator sign-in fails or /auth2/.well-known/openid-configuration advertises an http:// issuer and endpoints. Diagnosis. TLS terminates at the ingress controller, but the authentication workloads are not honoring the forwarded HTTPS scheme. Fix. Confirm the ingress sends X-Forwarded-Proto: https and that the application containers for auth-identity, auth-adminapi2, api-gateway, and api-registry have:
If your workload chart does not expose this environment setting, apply your approved admission mutation before restarting the affected pods. See Forwarded HTTPS scheme.

​
Prometheus scrape targets show as DOWN for /metrics-text

Symptom. In the in-mesh Prometheus, Vantage application targets show as DOWN. Diagnosis. On Istio, the Prometheus pod may be missing its sidecar or the Istio output certificates may not be written to /etc/prom-certs/. On Linkerd, a strict authorization policy may block plaintext scrapes from an unmeshed Prometheus.
Fix. With Istio, verify the Prometheus pod has three containers (prometheus, config-reloader, istio-proxy) and that the proxy.istio.io/config annotation with OUTPUT_CERTS: /etc/istio-output-certs is present. With Linkerd, inject the observability namespace or configure the ServiceMonitor and Prometheus with the client certificates required by your policy. See Monitoring with Prometheus.

​
Day-2 operations

​
After upgrade, the operator reports the new version but Vantage is still on the old one

Symptom. status.installedVantageVersion is unchanged after upgrading the Helm releases. Diagnosis. The vantage-operator and vantage-selfhosted releases may not be on the same new version, or ArgoCD may still be applying the per-component changes. Check both Helm releases and the ArgoCD application sync status. Fix. Upgrade both charts to the same version, then wait for the ArgoCD applications to finish syncing. If the operator surfaces a Degraded condition, follow the degradation reason to the failing component, resolve it, then annotate the Vantage resource with vantage.abbyy.com/reprocess to retry (Force a retry). See Upgrading.

​
What’s next

Architecture

What the operator does on every reconcile, useful context when diagnosing.

Known Limitations

Current constraints that shape what is and is not actually a bug.