Dynamic modules

Envoy has support for loading shared libraries at runtime to extend its functionality. In Envoy, these are known as “dynamic modules.” More specifically, dynamic modules are shared libraries that implement the ABI written in a pure C header file. The ABI defines a set of functions that the dynamic module must implement to be loaded by Envoy. Also, it specifies the functions implemented by Envoy that the dynamic module can call to interact with Envoy.

Implementing the ABI from scratch requires an extensive understanding of the Envoy internals. For users, we provide an official SDK that abstracts these details and provides a high-level API to implement dynamic modules. The SDK is currently available in C++, Go, and Rust. In theory, any language that can produce a shared library can be used to implement dynamic modules. Future development may include support for other languages.

Currently, dynamic modules are supported at the following extension points:

As a bootstrap extension (configuration).
As a cluster.
As a listener filter.
As a UDP listener filter (configuration).
As an access logger (configuration).
As a formatter.
As a stats sink (configuration).
As a network filter.
As an HTTP filter.
As an HTTP matching data input.
As an input matcher.
As a TLS certificate validator.
As a transport socket.
As a load balancing policy.
As an upstream HTTP TCP bridge.
As a tracer.
As a health checker (configuration).
As a cluster specifier (configuration).
As a DNS resolver (architecture). The Hickory resolver is implemented as a builtin dynamic module; the ABI and SDKs also support custom DNS resolvers.

There are a few design goals for the dynamic modules:

Performance: The dynamic modules should have minimal overhead compared to the built-in C++ extensions. For example, the dynamic modules are able to access HTTP headers as well as body without copying them unlike any other extension mechanisms.
Ease of Use: The SDK should provide a high-level API that abstracts the details of the Envoy internals.
Flexibility: The dynamic modules should be able to implement any functionality that can be implemented by the built-in C++ extensions without performance penalty. This is work in progress and many features are not yet available.

Compatibility

Since a dynamic module is loaded at runtime, it must be ABI-compatible with the Envoy binary that loads it.

Envoy’s dynamic modules have stricter compatibility requirements than Envoy’s other extension mechanisms, such as Lua, Wasm or External Processor. Stabilizing the ABI is challenging due to the way the ABI needs to be tightly coupled to Envoy’s internals.

Currently, we guarantee forward compatibility within one version: a dynamic module built with the SDK for Envoy version X.Y will work with Envoy versions X.Y and X.(Y+1). Breaking changes to the ABI may occur in later versions.

To ensure compatibility, it is recommended to rebuild your dynamic modules with the SDK matching your target Envoy version in a timely manner.

Module discovery

A dynamic module is referenced by its name as in the configuration API. The name is used to search for the shared library file in the search path. The search path is configured by the environment variable ENVOY_DYNAMIC_MODULES_SEARCH_PATH. The actual search path is ${ENVOY_DYNAMIC_MODULES_SEARCH_PATH}/lib${name}.so. If the environment variable is not set, the current working directory is used instead. After searching in the specified search path, the standard library paths such as LD_LIBRARY_PATH and /usr/lib are searched as well following the behavior of dlopen(3).

For example, when the name my_module is referenced in the configuration and ENVOY_DYNAMIC_MODULES_SEARCH_PATH is set to /path/to/modules, Envoy will first look for /path/to/modules/libmy_module.so, then $LD_LIBRARY_PATH/libmy_module.so, and finally /usr/lib/libmy_module.so, etc.

Safety

The dynamic modules should be used under the assumption that all modules are fully trusted and have the same privilege level as the main Envoy program. Since these modules run in the same process as Envoy, they can access all memory and resources available to the main process. This makes it unfeasible to enforce security boundaries between Envoy and the modules, as they share the same address space and permissions. It is essential that any dynamic module undergo thorough testing and validation before deployment just like any other application code.

Error handling (Rust SDK)

The Rust SDK provides an optional CatchUnwind wrapper that can be used to wrap filter implementations. When a wrapped callback panics, the SDK logs the panic payload and returns a fail-closed default:

HTTP request-path callbacks send a 500 response and return StopIteration.
HTTP response-path callbacks reset the stream and return StopIteration.
Network filter callbacks close the connection and return StopIteration.
Listener filter callbacks close the socket and return StopIteration.

The SDK always guards each callback at the FFI boundary so a panic can never unwind into Envoy and corrupt the process, regardless of whether CatchUnwind is used. The wrapper adds graceful filter-level teardown on top of that guard. The affected request or connection is terminated. Other traffic is unaffected.

Getting started

We have a dedicated repository for the dynamic module examples to help you get started. The repository is available at envoyproxy/dynamic-modules-examples

Statistics

All dynamic-module extension types emit the following statistics in the shared dynamic_modules. namespace. These stats track failures encountered while loading the extension’s configuration. Each one is tagged with config_name, set to the configured name of the dynamic-module extension instance — for example the filter_name for the HTTP filter, transport_socket_name for the transport socket, lb_policy_name for the load-balancing policy, tracer_name for the tracer or cluster_name for the cluster (default if the extension has no per-instance name, as for the UDP listener filter).

Name	Type	Description
module_load_error	Counter	Total dynamic modules that could not be loaded (missing or invalid module source, `dlopen` failure, by-name lookup miss, or a required ABI symbol could not be resolved).
config_init_error	Counter	Total configurations that failed to initialize after the module loaded successfully (the module rejected or failed to parse the supplied configuration).
remote_fetch_error	Counter	Total failures fetching or loading a remote module source, including rejected cache misses when `nack_on_cache_miss` is set. Only the HTTP filter supports remote module sources.
per_route_config_error	Counter	Total per-route configurations that failed to load or initialize. Only emitted by the HTTP filter.

In addition to the counters above, a module may define its own custom metrics. These are emitted under the configurable metrics_namespace (dynamicmodulescustom by default), separately from the dynamic_modules. namespace above.

Logging

When a dynamic module cannot be loaded, Envoy emits an error log on the dynamic_modules logger with the form Unable to load dynamic module <module>: <reason>. This is logged from the module loader itself, so it is emitted for every extension type, including extension points that have no factory context and therefore cannot increment the module_load_error counter. It is also the only signal when a module referenced from the bootstrap configuration fails to load, because Envoy exits before its statistics can be scraped.

Secrets

An HTTP filter module can subscribe to generic secrets rather than carrying credentials in its own configuration, so that secrets stay out of the module configuration and are rotated by Envoy.

A subscription is created while the filter configuration is being loaded, and is identified by a name plus an optional ConfigSource, serialized as JSON, that the module passes to Envoy:

Without a config source, the name refers to a secret in static_resources.secrets.
With a config source, the name is the resource requested over SDS. Identical subscriptions are shared across filters, and the value the module reads is updated whenever the SDS server pushes a new version. On the downstream HTTP filter chain the subscription also participates in initialization, so the listener does not start serving until the secret has been delivered.

Reading a secret returns the value for the current worker thread, so a rotation is transparent to the module. The returned bytes are owned by Envoy and are only valid for the duration of the event hook, so a module that needs to retain a secret must copy it.