> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abbyy.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Autoscaling with KEDA
> How Vantage 3.0 self-hosted uses KEDA, Prometheus, and Kubernetes HPAs to scale supported workloads.
KEDA is the optional, recommended autoscaling engine for Vantage 3.0 self-hosted. When you enable it, the Vantage component charts create `ScaledObject` resources for supported workloads. KEDA evaluates their triggers and manages Kubernetes HorizontalPodAutoscaler (HPA) resources that scale the workloads up and down.
The Vantage workload charts require KEDA 2.17.x. Use KEDA 2.17.3. KEDA 2.18 and later cannot process some `ScaledObject` fields generated by the current Vantage charts.
## How autoscaling works
| Stage | Component | Responsibility |
| ----- | ------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| 1 | Vantage workloads | Consume CPU and expose application metrics such as queue length and active worker threads. |
| 2 | Kubernetes resource metrics and Prometheus | Supply CPU utilization and Vantage application metrics to the configured triggers. |
| 3 | KEDA operator | Watches the chart-generated `ScaledObject` resources and evaluates their triggers. |
| 4 | Kubernetes HPA | Uses the replica recommendation produced for each metric and selects the largest recommendation. |
| 5 | Kubernetes scheduler | Places the requested replicas on eligible nodes with available resources. |
KEDA changes workload replica counts. It does not add Kubernetes nodes, increase quotas, or make an ineligible node eligible for a workload. Coordinate KEDA with your node autoscaler or other capacity-management process.
## Scaling strategies
The generated `ScaledObject` resources use different trigger types for different workload characteristics.
| Workload pattern | Trigger | Why it is used |
| -------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------- |
| Synchronous APIs and orchestration services | CPU utilization | Request load generally appears as CPU consumption while the service handles traffic. |
| Asynchronous processing and TechCore workers | Prometheus queue and worker metrics | Backlog and worker activity indicate pending work more directly than CPU utilization. |
The generated thresholds, replica boundaries, queries, and HPA behavior differ by workload and Vantage chart version. Treat the generated `ScaledObject` resources in your installation as the source of truth.
### CPU utilization triggers
CPU triggers compare a workload's average CPU utilization with its configured target. CPU utilization is calculated relative to the workload's CPU requests, so the Kubernetes resource metrics API must be healthy and the workload must retain valid CPU requests.
Confirm that resource metrics are available:
```bash theme={null}
kubectl top nodes
kubectl top pods -n $install_namespace
```
### Prometheus triggers
Asynchronous worker `ScaledObject` resources use Vantage application metrics from Prometheus. The principal signals are:
```promql theme={null}
application_worker_priority_queue_length{namespace=""}
application_worker_active_threads{namespace=""}
```
* `application_worker_priority_queue_length` reports work waiting for a worker type.
* `application_worker_active_threads` reports active processing threads. Some generated worker configurations use it as a second Prometheus trigger alongside queue depth: it scales the worker up preemptively when no processing threads remain free, so a newly arriving task is picked up immediately instead of waiting for a queue backlog to accumulate.
The chart-generated queries add workload-specific service and worker selectors and may use rolling time windows. Do not replace them with the two broad inspection queries above.
Some `ScaledObject` resources contain more than one trigger. Any trigger can request a scale-out, and the HPA uses the largest replica recommendation. When a workload scales unexpectedly, inspect every trigger rather than only the first one.
### Scale worker types in proportion to your skills
TechCore processing is divided into worker types, such as OCR, Extraction, and Classification, and each worker type scales independently on its own queue. Your deployed skills determine which worker types carry the most work: an OCR-heavy skill mix drives demand for OCR workers, while an extraction-heavy mix drives demand for Extraction workers.
Because documents move through these worker types in stages, capacity imbalance between stages appears as waiting rather than as errors. If one stage is under-provisioned relative to the stage that feeds it (for example, ample OCR capacity but too few Extraction workers), documents clear the first stage quickly and then pause for several minutes while the downstream queue drains and its workers scale out. Watch per-worker-type queue depth during a representative load test with your production skill mix, and discuss adjustments to the relative replica boundaries with your ABBYY account team.
## Prerequisites
Before enabling KEDA, confirm that:
1. KEDA 2.17.3 is installed and its operator and metrics API server are healthy.
2. The Kubernetes resource metrics API returns CPU data.
3. Prometheus is reachable at `http://prometheus-operated.observability.svc.cluster.local:9090`.
4. The Vantage `ServiceMonitor` exists and Prometheus reports the Vantage `/metrics-text` targets as `UP`.
5. Prometheus returns current values for the Vantage worker metrics.
6. The cluster has enough schedulable capacity for the additional replicas, including nodes labeled for TechCore workers.
The Prometheus service name, namespace, and port in item 3 form a fixed endpoint for the generated `ScaledObject` resources.
Use these checks before continuing:
```bash theme={null}
kubectl get deployment --all-namespaces | grep keda-operator
kubectl -n observability get service prometheus-operated
kubectl -n $install_namespace get servicemonitor
```
In Prometheus, confirm that these queries return series for your install namespace:
```promql theme={null}
application_worker_priority_queue_length{namespace=""}
application_worker_active_threads{namespace=""}
```
For Prometheus installation and service-mesh requirements, see [Monitoring with Prometheus](/vantage/self-hosted/v3.0/monitoring/prometheus).
## Enable autoscaling
Enable KEDA in the Vantage configuration and enable the chart-created `ServiceMonitor` through the separate top-level `observability` stanza:
```yaml theme={null}
vantage:
keda:
enabled: true
observability:
prometheus:
namespace: observability
serviceMonitor:
create: true
```
The `observability` stanza is not part of `vantage` or the `Vantage` custom resource. It controls monitoring resources rendered directly by the `vantage-selfhosted` chart.
Apply the values through your existing `vantage-selfhosted` Helm release. ArgoCD creates the component applications and their `ScaledObject` resources asynchronously.
## Verify autoscaling
List the generated resources:
```bash theme={null}
kubectl -n $install_namespace get scaledobjects.keda.sh
kubectl -n $install_namespace get hpa
```
Wait for the `ScaledObject` resources to become ready:
```bash theme={null}
kubectl -n $install_namespace wait \
--for=condition=Ready scaledobjects.keda.sh \
--all \
--timeout=5m
```
Inspect an individual resource and its HPA:
```bash theme={null}
kubectl -n $install_namespace describe scaledobject
kubectl -n $install_namespace get scaledobject -o yaml
kubectl -n $install_namespace describe hpa
```
Use the status carefully:
* `Ready=True` means KEDA accepted the trigger configuration.
* `Active=True` means at least one trigger is currently requesting scaling activity.
* `Active=False` can be normal when there is no traffic or queued work.
* An HPA metric value of `` indicates that Kubernetes cannot obtain a current value for that metric.
During a controlled workload test, watch the workload, HPA, and node capacity together:
```bash theme={null}
kubectl -n $install_namespace get deployment,hpa,pod --watch
```
Confirm that requested replicas become `Running`. A rising HPA desired count with pods remaining `Pending` is a capacity or scheduling problem, not a KEDA trigger problem.
## Understand the generated behavior
The Vantage workload charts supply workload-specific settings for:
* Trigger thresholds and Prometheus queries.
* Minimum and maximum replica boundaries.
* Scale-up rate and stabilization behavior.
* Scale-down rate and stabilization behavior.
The configuration is designed to scale up quickly when backlog appears and to scale down cautiously through a stabilization window: a cool-down that keeps replicas running until the queue has stayed quiet for a set period. Model and training workers use a longer cool-down than lightweight API services because removing their replicas discards cached models, and reloading those models makes the next task noticeably slower. Treat the window durations in your generated resources as reference values for that chart version, not tuning targets.
Exact values are product configuration, not a universal capacity recommendation. They can change with Vantage chart versions as workload characteristics and validation data evolve.
Do not copy `ScaledObject` manifests or thresholds from another environment, and do not edit generated resources directly. ArgoCD can restore the chart-rendered configuration during reconciliation or upgrade. Use only supported Vantage chart values; contact your ABBYY account team before applying additional autoscaling overrides.
## Plan cluster capacity
Autoscaling cannot compensate for an undersized or constrained cluster. Plan for:
* The maximum document arrival rate and acceptable queue wait time.
* Average and high-percentile processing duration for representative documents and skills.
* Pod startup and model warm-up time.
* CPU and memory requests for every concurrently scaling workload.
* Node-autoscaler provisioning time and maximum node-pool size.
* Namespace quotas, cluster limits, and registry pull throughput.
* TechCore node labels, taints, accelerators, and training-worker isolation.
* Failure tolerance when a node or availability zone is unavailable.
Measure queue depth, processing latency, pod startup time, HPA desired replicas, and pending pods during a representative load test. Use those observations when discussing environment-specific tuning with ABBYY.
## Troubleshooting
| Symptom | Likely area to inspect |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| No `ScaledObject` resources appear | Confirm `vantage.keda.enabled: true` and inspect the generated ArgoCD applications. |
| `Ready=False` or `ScaledObjectCheckFailed` | Confirm KEDA 2.17.3 and inspect the `ScaledObject` events for an unsupported trigger field. |
| HPA metrics show `` | Check the Prometheus service, `ServiceMonitor`, scrape targets, mesh policy, and query results. |
| HPA requests more replicas but pods remain `Pending` | Check node capacity, quotas, resource requests, TechCore labels, taints, and persistent-volume scheduling. |
| A workload scales when its queue appears quiet | Inspect all triggers; another Prometheus or CPU trigger may be active. |
| Scaling behavior changes after an upgrade | Compare the generated `ScaledObject` with the versioned chart configuration and confirm both Vantage charts use the same version. |
For detailed corrective steps, see [Troubleshooting KEDA and Prometheus](/vantage/self-hosted/v3.0/troubleshooting#keda-and-prometheus).
## What's next
Configure scraping and verify the metrics used by KEDA.
Review KEDA, Prometheus, and cluster-capacity requirements.
Diagnose unhealthy ScaledObjects and unavailable metrics.
Check supported versions before upgrading KEDA or Vantage.