> ## Documentation Index
> Fetch the complete documentation index at: https://docs.abbyy.com/llms.txt
> Use this file to discover all available pages before exploring further.

# System monitoring and bottleneck detection

> Monitor ABBYY FlexiCapture servers and detect bottlenecks using Windows Performance Monitor counters for memory, CPU, disk, network, and per-component metrics.

System monitoring includes:

* Document processing monitoring via the Administration and Monitoring Console
* Hardware monitoring for each FlexiCapture server component using various Windows Performance Monitor counters

You can perform hardware monitoring of the entire System using the free [Web Performance Monitor application](https://www.iis.net/downloads/community/2007/01/web-performance-monitor), the more powerful Microsoft System Center Operations Manager, or similar tools.

## Key counters for every computer

These are the key parameters to keep track of on every computer, no matter its role inside FlexiCapture.

<Note>
  A shortage of just one resource may lead to an overload on any component. For example, a shortage of RAM may lead to very intensive use of the hard drive. This is why the order in which you investigate parameters to find a bottleneck matters. Follow the order below.
</Note>

### Memory

When the `Memory\Available Bytes` counter (the memory not occupied by running processes and a hard disk cache) is *constantly low*, while the `Memory\Pages/sec` counter (the number of memory pages requested from or written to the hard drive to release more RAM) is *constantly changing*, it is likely that the computer has insufficient RAM.

The `Process(<All instances>)\Working Set` counters show the number of memory pages allocated to each process.

* Each process may withhold a large amount of memory and the total available memory may be low, but this does not mean a memory shortage in the System.
* *However, if you see that several processes enlarge their working sets while others slightly shrink them, the available amount of memory is low, and the `Pages/sec` counter constantly spikes, this is likely to be a RAM bottleneck.*

<Note>
  32-bit processes can't allocate more than 2 GB of RAM even if there is a lot of available RAM in the System.
</Note>

### CPU

When the `Processor(_Total)\% Processor Time` counter (the percentage of time the processor is busy) *shows more than 80% for significant periods of time*, and the `System\Processor Queue Length` counter (the number of threads in the CPU queue) exceeds double the number of CPUs, the CPU is most likely causing a bottleneck.

The `Process(<All instances>)\% Processor Time` counter helps determine which processes are "eating up" the CPU time.

### Hard disk

When checking the hard disk, make sure the System has enough RAM (see [Memory](#memory) above).

The `LogicalDisk(<All instances>)\Free Megabytes` counter shows the free space on a logical disk. If the free space is not enough, System performance drops significantly.

When the `PhysicalDisk(<All instances>)\% Disk Time` counter (the percentage of time the disk spends processing read-and-write requests) *shows more than 90%*, and the `PhysicalDisk(<All instances>)\Avg. Disk Queue Length` counter (the average number of requests in the queue and in progress during the monitoring period) *constantly shows more than 2 per hard drive thread*, the hard disk is probably causing a bottleneck.

### Network

When the `Network Interface(<All instances>)\Output Queue Length` counter (the number of outgoing network packets in a queue) *constantly shows more than 2*, the network adapter is most likely waiting for the connection, delaying requests from the server.

When the `Network Interface(<All instances>)\Packets Outbound Discarded` counter *is constantly growing*, the channel is so overloaded that the buffer of the network adapter cannot handle all of the outgoing requests.

When the `Network Interface(<All instances>)\Bytes Total/sec` counter (the amount of information passing through the NIC) *reaches 65% or more* of the `Network Interface(<All instances>)\Current Bandwidth` counter (the available NIC bandwidth), use a channel with a higher bandwidth, or segment the network to minimize conflicts in the channel.

## Counters to watch by component

### Application Server

The Application Server is a web service on IIS served by `w3wp.exe` processes. The CPU time and amount of memory used by instances of these processes should be supervised. The data connection network (DCN) load on a computer with the Application Server is worthy of special attention.

Use these counters to keep track of IIS loads:

* `Web Service(Default Web Site)\Bytes Received/sec`
* `Web Service(Default Web Site)\Bytes Sent/sec`

When they display more than 65% of the available DCN bandwidth, use any of these solutions:

* Increase the bandwidth
* Segment the network
* Add more computers running IIS, and balance the loads

When the value of the `W3SVC_W3WP(_Total)\Active Threads Count` counter (the number of threads servicing requests) reaches the value of the `W3SVC_W3WP(_Total)\Maximum Threads Count` counter (the maximum number of threads available for servicing), IIS is heavily overloaded.

Use these counters to keep track of current activity spikes compared with previous activity:

* `W3SVC_W3WP(_Total)\Active Requests` (the number of active requests)
* `W3SVC_W3WP(_Total)\Requests/sec` (the request processing rate)
* `Web Service(Default Web Site)\Current Connections` (the number of active connections between clients and the web service)

### Processing Server

The Processing Server is a Windows service – the `FlexiBrSvc.exe` process.

`FlexiCapture(Processing Server)\Cores Count` indicates the total number of processing CPU cores on all Processing Stations in the **Started** state.

`FlexiCapture(Processing Server)\Free Cores` indicates the total number of available processing CPU cores that are not processing anything. This counter helps determine whether there are enough processing CPU cores in ABBYY FlexiCapture.

Some cores are always free while there is a queue at automatic processing stages. This may happen because:

* Some Processing Stations are configured to process a dedicated type of task, and when they are free, there is insufficient processing power for other types of tasks.
* The Processing Server experiences a bottleneck:
  * Check for a shortage of RAM, CPU, hard drive, or network resources on the Processing Server machine.
  * Enlarge the size of the task, as processing cores process tasks too quickly and the Processing Server fails to distribute tasks between them (see [How to calculate the number of Processing Stations](/flexi-capture/performance-guide/pg-calculate#2) for more details).

`FlexiCapture(Processing Server)\Pending Tasks` indicates the number of tasks requested from the Application Server but not assigned to any of the Processing Stations. Such tasks appear in the Processing Server Monitor with the **Pending** status.

<Note>
  The whole queue of tasks waiting for processing can be viewed only in the Administration and Monitoring Console. The value of `Pending Tasks` should not exceed double the number of available cores (`Cores Count`). If the number of pending tasks is constantly growing, this means either that particular processing options on some Stations are disabled, or that there are communication errors between a Station and the Server: the Server considers some Stations to be live but cannot deliver a task to them.
</Note>

### Licensing Server

The Licensing Server is a Windows service – the `LicensingService.exe` process.

You can use the COM object of the Licensing Server to monitor license statuses, but in most cases it provides no useful data for System performance.

Keep an eye on memory consumption – it may grow with the number of concurrent clients in the System. Consider adding more Licensing Servers when memory consumption is high and you need to support more clients.

### Database Server

In addition to the standard System monitoring counters (*see above*), you can also use counters that show data on a specific database. For details, see the documentation for your Database Server.

### FileStorage

For a hard disk used as FileStorage, use the standard System monitoring counters (*see above*). If you are using a SAN or NAS, see your hardware documentation for details.

### Processing Stations

Use the standard System counters to monitor Processing Stations.
