Pending request queueing

When no upstream connection is immediately available, a connection pool may queue a request while it establishes a connection or waits for connection capacity. The maximum pending requests circuit breaker limits how many requests may wait. A queue policy controls the order in which those requests are selected when capacity becomes available; it does not change the circuit breaker limit.

The policy is configured with Cluster.queuing_policies. If no policy is configured, Envoy uses FIFO ordering.

Queue policies

FIFO

The FIFO queue policy selects the oldest pending request first.

Adaptive LIFO

The adaptive LIFO queue policy uses FIFO ordering until the queue size reaches the configured high watermark. It then selects the most recently queued request until the queue size reaches or falls below the low watermark. The watermarks are absolute request counts applied independently to each connection pool’s pending request queue. Separate watermarks prevent the policy from repeatedly switching order near the overload boundary.

Adaptive LIFO prioritizes newer requests during overload. Older requests may wait longer, and under sustained overload they may time out before being selected.

The following cluster fragment configures each connection pool to enter LIFO mode when it has 80 pending requests and return to FIFO mode when it has 60 or fewer pending requests:

queuing_policies:
  pending_rq_policy:
    name: envoy.queue_policy.adaptive_lifo
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.queue_policy.adaptive_lifo.v3.AdaptiveLifoQueuePolicyConfig
      high_watermark: 80
      low_watermark: 60

Observability

The upstream_rq_pending_active gauge reports the number of requests currently waiting for a connection pool connection. For queue policies that report an overloaded state, such as adaptive LIFO, upstream_queue_overloaded reports the number of connection pool pending request queues currently overloaded. Both are documented in the cluster statistics.