Performance Tuning

The default settings of the Data Hub allow for very high throughput while balancing system resources (CPU, memory, disk, network) on most hosts, both during normal delivery and during recovery scenarios (e.g. after a destination that was offline comes back online). Changing these settings can have unexpected consequences, so we recommend leaving these settings at their default values unless you first consult with PreEmptive support about any issues you are having.

The most common scenarios where you might need to make changes are for a slow network connection or for a destination that is getting overwhelmed when it comes online after an outage.

For performance considerations before installing, see High-volume Performance Considerations.

Tasks

The Dispatch Service operates in a highly-concurrent manner, using a combination of .NET threads and .NET tasks. At the application level, these concurrent activities are rolled up into a single concept (in e.g. the Data Hub configuration) called tasks. There can be many concurrent tasks, each responsible for reading a batch of messages from a source queue and dispatching them appropriately.

There are two types of tasks:

An endpoint task processes messages queued in the endpoint queue by the Endpoint Web Service, evaluating include and exclude criteria and dispatching each message to all relevant destinations.
A destination-specific task processes messages that were previously queued by the Dispatch Service in that destination's error or offline queue, re-attempting dispatch of each message to its destination.

At any given time, each task holds at most one outgoing HTTP connection to a particular destination.

While a task is processing, its messages are not actually removed from the source queue. Instead, they are reserved by the task and marked unacknowledged in RabbitMQ. When the task has processed all of its messages for all destinations, these messages are removed from their source queue. If the Dispatch Service is stopped or interrupted during processing, the set of messages will go back onto their source queue, preventing data loss.

The number of tasks in use at a time is limited to prevent resource overuse and contention. The number of messages that each task can process is also limited. These limits can be configured, to tune throughput and concurrency.

General Considerations

The Data Hub operates the most efficiently when all (durable) destinations are accepting all messages, so that no messages are being queued. In this state, the Data Hub should keep the endpoint queue near-empty, and CPU, memory, disk, and network overhead should be small.

A large number of error responses, or an offline destination, will affect performance upon recovery: in addition to new incoming messages, the Dispatch Service (and the destination) must handle the backlog of previously-queued messages. A similar issue occurs if the Dispatch Service is itself stopped for an extended period, as the endpoint queue will need to be cleared when the service resumes. (These scenarios also lead to greater disk usage.) In these scenarios, the Data Hub will attempt to deliver the backlog as fast as it possibly can, typically throttled by CPU or network bandwidth until the recovery is complete.

Each new message, when it is first processed by the Dispatch Service, must be evaluated to determine which destination(s) it will be dispatched to. Thus, high levels of complexity in include and exclude criteria may cause high CPU use.

Destination latency can impact performance, as the connection attempt will continue until either a response is received or the Delivery Timeout elapses, marking the destination offline. As all messages in a task must be processed for all relevant destinations before the task is complete, and because endpoint tasks are not destination-specific, a latent destination may slow throughput of new messages to other destinations.

Large message sizes can also impact performance as well as increase the memory usage by IIS, the Dispatch Service and RabbitMQ. Overall memory usage by the Data Hub can be controlled by adjusting the Dispatch Service Concurrency Settings.

In addition to performance considerations on the Data Hub, also consider the impact on destinations. As the Data Hub performs no processing on the actual bodies of messages, it may dispatch messages faster than a destination (or the connection to the destination) can handle. If the destination either begins refusing connections or does not respond in the Delivery Timeout, the destination will be marked offline. While this will alleviate pressure on the destination temporarily, upon a successful retry, the destination will now need to deal with both new incoming messages and the backlog accrued.

Dispatch Service Concurrency Settings

Specific concurrency and delivery settings can be configured for the Dispatch Service in the <appSettings> section of [Application folder]\DispatchService\HubDispatchService.exe.config. You must restart the Dispatch Service for the settings to take effect.

The settings are as follows:

DeliveryTimeout
- The time span to wait for a destination to respond before considering the destination to be offline.
- Too low of a value may lead to incorrectly marking a destination as offline when it is still operational. Too high of a value will lead to unnecessarily lengthy use of an outgoing connection when a destination is not responding.
- Expressed in the time formats allowed by similar Dispatch.config settings.
- Default: 10 seconds.
SecondsBeforeDispatch
- The timer, in seconds, for starting a new endpoint task.
- Too low of a value may lead high CPU use. Too high of a value will lead to slowed throughput from the endpoint queue.
- Expressed solely in seconds.
- Default: 0.01 seconds.
MaxMessagesPerTask
- The maximum number of messages to process per task.
- Too low of a value will reduce maximum throughput. Too high of a value will lead to longer-running tasks, preventing newer messages from being processed as quickly. High values will also result in higher memory usage by the Dispatch Service, especially with larger message sizes.
- Default: 25.
MaxEndpointTasks
- The maximum number of tasks allowed to be processing messages from the endpoint queue at any given time.
- Too low of a value will cause slowed throughput from the endpoint queue. Too high of a value may lead to resource contention among the tasks. High values will also result in higher memory usage by the Dispatch Service, especially with larger message sizes.
- Default: 8.
MaxTasksPerDestination
- The maximum number of tasks allowed to process messages from the offline and error queues of a specific destination at any given time.
- Too low of a value will reduce throughput from the destination-specific queues. Too high of a value may lead to resource contention among the tasks. High values will also result in higher memory usage by the Dispatch Service, especially with larger message sizes.
- Default: 4.

Maximum Outgoing Connections

The maximum number of outgoing requests to a given destination (from the Dispatch Service) can be configured. Higher values for this setting may increase performance, but can also lead to higher load on destinations. Lowering this value might help prevent timeouts from destinations that are being overwhelmed after coming back online.

To change this value:

Open [Application folder]\DispatchService\HubDispatchService.exe.config.
In the <system.net> section, locate the <connectionManagement> subsection.
Within this subsection, modify the <add address="*"> tag's maxconnection attribute appropriately.
Save the file.
Restart the Dispatch Service.

The default value, 24, was chosen based on the total number of connections that would be needed (in the worst case) by summing the default MaxEndpointTasks and MaxTasksPerDestination values, below. This is because each task uses, at most, one outgoing connection to a particular destination at a time. To prevent resource contention, we recommend keeping this pattern as you adjust any of these three values:

maxconnection = MaxEndpointTasks + MaxTasksPerDestination

Load Balancer Support

To balance a large load of incoming messages, multiple Data Hub instances can be configured to accept requests in a round-robin fashion using third-party load balancers. However, there are some considerations to address when doing so: