Extension Toolkit Reference

Introduction

The purpose of GetEnvoy Extension Toolkit is to help developers curious about the extensibility of Envoy to get up and running in seconds.

By using GetEnvoy Extension Toolkit, you:

  • will be able to start from a working and representative example;
  • will have effective development workflow set up from the beginning;
  • will leverage best practices and avoid common pitfalls by default.

Supported extension types and languages

Supported extension types:

  • HTTP Filter
  • Network Filter
  • Access Logger

Supported languages:

  • Rust

Development Flow

getenvoy CLI provides a group of commands that automate a familiar development flow:

init => build => test => run

Step Role
init scaffold a new extension
build build a WebAssembly module (*.wasm file)
test run language-specific unit tests
run launch the extension in Envoy using an example setup

Walkthrough

Let’s walk through the development flow of a WebAssembly-based Envoy extension using the GetEnvoy Extension Toolkit.

1. Pre-requirements

  1. πŸ’» Install GetEnvoy.

    πŸ’‘ getenvoy CLI will keep its internal files (such as caches and downloaded artifacts) at $HOME/.getenvoy.

  2. πŸ’» Install Docker.

    πŸ’‘ getenvoy CLI uses Docker to run language-specific build tools inside containers, e.g. Cargo (Rust).

2. Scaffold a new HTTP Filter extension

πŸ’» To walk through the interactive wizard, run:

$ getenvoy extension init

πŸ’‘ Alternatively, to skip the wizard, provide the arguments on the command line, e.g.:

$ getenvoy extension init \
    --category envoy.filters.http \
    --language rust \
    --name me.filters.http.my_http_filter \
    my_http_filter

βœ… getenvoy will scaffold a new extension workspace in the target directory:

$ tree -a my_http_filter
 
my_http_filter
β”œβ”€β”€ .cargo
β”‚   └── config
β”œβ”€β”€ .getenvoy
β”‚   └── extension
β”‚       └── extension.yaml
β”œβ”€β”€ .gitignore
β”œβ”€β”€ Cargo.toml
β”œβ”€β”€ README.md
β”œβ”€β”€ src
β”‚   β”œβ”€β”€ config.rs
β”‚   β”œβ”€β”€ factory.rs
β”‚   β”œβ”€β”€ filter.rs
β”‚   β”œβ”€β”€ lib.rs
β”‚   └── stats.rs
└── wasm
    └── module
        β”œβ”€β”€ Cargo.toml
        └── src
            └── lib.rs

πŸ“šFor further details and usage examples refer to the documentation of getenvoy extension init.

Generated files

πŸ’‘ At a high level, extension workspace consists of 2 groups of files:

  1. programming language-specific source code, e.g. Rust sources, Cargo config, etc
  2. GetEnvoy Extension Toolkit-specific metadata, i.e. all files under .getenvoy/extension/
Source code

πŸ’‘ Rust version of init templates is based on Envoy SDK for Rust, which gives WebAssembly-based extensions a model similar to their native counterparts, including concepts such as Extension, Extension Factory, Extension Config, Extension Stats, etc.

Metadata files

πŸ’‘ Metadata files, such as .getenvoy/extension/extension.yaml, drive behaviour of various getenvoy commands.

Metadata files can be edited manually and should be kept under source control along with the extension source code.

3. Build the extension

πŸ’» To build a WebAssembly module (*.wasm file) loadable by Envoy, run:

$ getenvoy extension build

βœ… getenvoy will internally run a language-specific build tool to generate a WebAssembly module (extension.wasm):

$ tree target/getenvoy

target/getenvoy
└── extension.wasm

πŸ’‘ To support multiple programming languages and be able to work in arbitrary user environments, getenvoy commands by default leverage Docker containers to run language-specific tools.

E.g., in the case of Rust, getenvoy uses docker.io/getenvoy/extension-rust-builder image to run cargo build command.

πŸ’‘ If your extension requires a non-standard build image or extra container options, you can override the defaults either temporarily (via command line options) or permanently (via a respective metadata file).

βœ… on first use of the build command, getenvoy will generate a toolchain metadata file to let you override the default build behaviour:

$ tree .getenvoy/extension/toolchains

.getenvoy/extension/toolchains
└── default.yaml

πŸ“šFor further details and usage examples refer to the documentation of getenvoy extension build.

4. Run unit tests

πŸ’» To execute programming language-specific unit tests, run:

$ getenvoy extension test

βœ… getenvoy will pipe output of the language-specific test framework:

running 1 test
test tests::should_initialize ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

πŸ’‘ As mentioned earlier, getenvoy by default leverages Docker containers to run unit tests.

E.g., in the case of Rust, getenvoy uses docker.io/getenvoy/extension-rust-builder image to run cargo test command.

πŸ“šFor further details and usage examples refer to the documentation of getenvoy extension test.

5. Run the extension in Envoy (the “easy” way)

πŸ’» To run the extension in Envoy, execute:

$ getenvoy extension run

βœ… getenvoy will download Envoy binary, generate a sample Envoy config and start the Envoy process in the foreground:

info	Envoy command: [$HOME/.getenvoy/builds/wasm/1.15/darwin/bin/envoy -c /tmp/getenvoy_extension_run732371719/envoy.tmpl.yaml]
...
[info][main] [external/envoy/source/server/server.cc:339] admin address: 127.0.0.1:9901
...
[info][config] [external/envoy/source/server/listener_manager_impl.cc:700] all dependencies initialized. starting workers
[info][main] [external/envoy/source/server/server.cc:575] starting main dispatch loop

πŸ“šFor further details and usage examples refer to the documentation of getenvoy extension run.

5+. Run the extension in Envoy (the “hard” way)

πŸ’‘ To get a better understanding of what getenvoy extension run does, we can also reproduce its steps manually:

  1. πŸ’» First, we need to build a WebAssembly module (*.wasm file) that will be loaded by Envoy:

     $ getenvoy extension build
    

    βœ… getenvoy will internally run a language-specific build tool to generate a WebAssembly module (extension.wasm):

     $ tree target/getenvoy
         
     target/getenvoy
     └── extension.wasm
    
  2. πŸ’» Next, we need to prepare Envoy config suitable for this particular extension:

     $ getenvoy extension examples add
    

    βœ… getenvoy will generate the default example setup:

     $ tree .getenvoy/extension/examples/
         
     .getenvoy/extension/examples/
     └── default
         β”œβ”€β”€ README.md
         β”œβ”€β”€ envoy.tmpl.yaml
         β”œβ”€β”€ example.yaml
         └── extension.json
    
  3. πŸ’» Next, we need to find out what version of Envoy the extension is compatible with:

     $ cat .getenvoy/extension/extension.yaml
    

    βœ… getenvoy records the compatible version of Envoy at a time of getenvoy extension init:

     # Runtime the extension is being developed against.
     runtime:
       envoy:
         version: wasm:1.15
    

    ⚠️ Since WebAssembly support in Envoy is still under active development, no assumptions can be made about API compatibility between various Envoy releases.

  4. πŸ’» Next, we need to download Envoy binary of that version:

     $ getenvoy fetch wasm:1.15
    

    βœ… getenvoy will download Envoy binary and cache it under $HOME/.getenvoy:

     $ tree $HOME/.getenvoy/builds
         
     $HOME/.getenvoy/builds
     └── wasm
      Β Β  └── 1.15
      Β Β      └── darwin
      Β Β          └── bin
      Β Β              └── envoy
    
  5. πŸ’» Finally, we need to wire together various configuration bits (such as, Envoy config, *.wasm file,
    extension-specific config) and start the Envoy process:

    $ getenvoy extension run
    

    βœ… getenvoy will generate the actual Envoy config (by resolving placeholders in the configuration template ) and start the Envoy process in the foreground:

    info	Envoy command: [$HOME/.getenvoy/builds/wasm/1.15/darwin/bin/envoy -c /tmp/getenvoy_extension_run732371719/envoy.tmpl. yaml]
    ...
    [info][main] [external/envoy/source/server/server.cc:339] admin address: 127.0.0.1:9901
    ...
    [info][config] [external/envoy/source/server/listener_manager_impl.cc:700] all dependencies initialized. starting workers
    [info][main] [external/envoy/source/server/server.cc:575] starting main dispatch loop
    

6. Make sample requests

πŸ’‘ Every example setup comes with instructions on how to use it.

πŸ’» Checkout README.md file included in the default example setup:

   $ open .getenvoy/extension/examples/default/README.md

βœ… In the case of a HTTP Filter, instructions will be similar to the following:

   ## How to use
   
   1. Make HTTP request
      ```shell
      curl http://0.0.0.0:10000
      ```
   2. Checkout `Envoy` stdout

πŸ’» Follow How to use instructions from README.md

βœ… HTTP Filter extension will log the following to the output of Envoy process:

my_http_filter: #2 new http exchange starts at 2020-07-01T18:22:51.623813+00:00 with config:
my_http_filter: #2 observing request headers
my_http_filter: #2 -> :authority: 0.0.0.0:10000
my_http_filter: #2 -> :path: /
my_http_filter: #2 -> :method: GET
my_http_filter: #2 -> user-agent: curl/7.64.1
my_http_filter: #2 -> accept: */*
my_http_filter: #2 -> x-forwarded-proto: http
my_http_filter: #2 -> x-request-id: 8902ca62-75a7-40e7-9b2e-cd7dc983b091
my_http_filter: #2 http exchange complete

7. Get familiar with the source code

πŸ’‘ As mentioned earlier, Rust version of init templates is based on Envoy SDK for Rust, which gives WebAssembly-based extensions a model similar to their native counterparts, including concepts such as Extension, Extension Factory, Extension Config, Extension Stats, etc.

Let’s take a closer look into the source code of a HTTP Filter extension.

File Role
Cargo.toml Rust package with the source code of HTTP Filter extension.
src/config.rs Types that represent extension’s configuration.
src/factory.rs Types that represent extension’s Factory.
src/filter.rs Types that represent extension itself (HTTP Filter).
src/lib.rs Rust library.
src/stats.rs Types that represent metrics collected by the extension.
wasm/module/Cargo.toml Rust package representing a WebAssembly module that bundles one or more extensions together.
wasm/module/src/lib.rs Entrypoint of a WebAssembly module (_start function).

Lifecycle of the WebAssembly module

  • Envoy extensions get distributed as WebAssembly modules (*.wasm files).
  • A single WebAssembly module can include multiple extensions bundled together.
  • Envoy starts loading a WebAssembly module as soon as it receives a Listener configuration that refers to that module.
  • Once a module is loaded into memory, Envoy will call the _start function to let the module initialize itself, which typically includes initializing static variables, configuring logging, registering extensions provided by the module, etc.
  • Envoy will unload the WebAssembly module as soon as the Listener that refered to it has been removed.

Lifecycle of the Extension Factory

  • Extension Factory gets created once, when Envoy is applying a new Listener configuration
  • After that, Envoy will call Extension Factory to pass it extension-specific configuration
  • Extension Factory can parse configuration (typically in JSON format) and initilize state that will be shared by all Extension instances

Lifecycle of the HTTP Filter

  • HTTP Filter in Envoy is a stateful object that is only processing a single HTTP request
  • Once Envoy receives a new HTTP request, it will call the Extension Factory to create a new HTTP Filter instance
  • Next, Envoy will start calling lifecycle callbacks defined on that HTTP Filter, e.g.
    • on request headers
    • on request body chunk
    • on response headers
    • on exchange complete
    • etc
  • Within the context of lifecycle callbacks, HTTP Filter can do the following:
    • suspend/resume further processing of that HTTP request
    • mutate headers/body/trailers of HTTP request and response
    • make auxiliary HTTP requests
    • record metrics
    • record extra data for inclusion into Access Log
    • etc
  • HTTP Filter instance will be destroyed once Envoy finishes processing the HTTP requests it is associated with

Metadata

πŸ’‘ GetEnvoy Extension Toolkit relies on metadata to drive various getenvoy commands.

Collectively, metadata are all the files under .getenvoy/extension/ directory in the extension workspace.

E.g.,

$ tree .getenvoy/extension/

.getenvoy/extension/
β”œβ”€β”€ examples
β”‚Β Β  └── default
β”‚Β Β      β”œβ”€β”€ README.md
β”‚Β Β      β”œβ”€β”€ envoy.tmpl.yaml
β”‚Β Β      β”œβ”€β”€ example.yaml
β”‚Β Β      └── extension.json
β”œβ”€β”€ extension.yaml
└── toolchains
    └── default.yaml

πŸ’‘ metadata files will typically be auto-generated on first use.

Extension descriptor

πŸ’‘ Extension descriptor represents meta information about the extension, such as extension name, type, programming language, etc.

Extension descriptor is generated automatically by getenvoy extension init.

E.g.,

kind: Extension

name: me.filters.http.my_http_filter

category: envoy.filters.http
language: rust

# Runtime the extension is being developed against.
runtime:
  envoy:
    version: wasm:1.15

Toolchain

πŸ’‘ Toolchain represents a configuration of language-specific build tools that get called internally by getenvoy extension build, getenvoy extension test, getenvoy extension clean, etc.

default toolchain is generated automatically on first use of getenvoy extension build and is stored under .getenvoy/extension/toolchains/, e.g.

$ tree .getenvoy/extension/toolchains/

.getenvoy/extension/toolchains
└── default.yaml
#
# Configuration for the built-in toolchain.
#
kind: BuiltinToolchain

#
# Configuration of the default build container.
#

## container:
##   # Builder image.
##   image: getenvoy/extension-rust-builder:0.2.0
##   # Docker cli options.
##   options: []

#
# Configuration of the 'build' command.
#
# If omitted, configuration of the default build container will be used instead.
#

## build:
##   container:
##     # Builder image.
##     image: getenvoy/extension-rust-builder:0.2.0
##     # Docker cli options.
##     options: []
##   output:
##     # Output *.wasm file.
##     wasmFile: target/getenvoy/extension.wasm

...

Example setups

πŸ’‘ Example setup represents an Envoy configuration that demonstrates the extension in action on real traffic.

Every extension can have multiple Example setups:

Example setups get stored under .getenvoy/extension/examples/, e.g.:

$ tree .getenvoy/extension/examples/

.getenvoy/extension/examples/
└── default
    β”œβ”€β”€ README.md
    β”œβ”€β”€ envoy.tmpl.yaml
    β”œβ”€β”€ example.yaml
    └── extension.json

where

File Description Purpose
README.md README file Describes individual Example setup, provides How To Use instructions
example.yaml Example descriptor Describes runtime requirements, e.g. a specific version of Envoy
envoy.tmpl.yaml Envoy bootstrap config Provides Envoy config that demoes extension in action
extension.json Extension config Provides configuration for extension itself

example.yaml

πŸ’‘ Example setup includes a descriptor file example.yaml that can be used to express runtime requirements specific to that example.

E.g.,

kind: Example

# Runtime required by the example.
runtime:
  envoy:
    version: wasm:1.15

envoy.tmpl.yaml

πŸ’‘ Example setup includes a template Envoy configuration that is stored in envoy.tmpl.yaml.

To the most part, it is a regular envoy.yaml bootstrap config file.

Additionaly, contents of envoy.tmpl.yaml is also allowed to use placeholders specific to GetEnvoy Extension Toolkit.

E.g., by default, envoy.tmpl.yaml will have the contents similar to the following:

#
# Example Envoy configuration.
#
admin: {{ .GetEnvoy.DefaultValue "admin" }}                 # notice use of the placeholder

static_resources:
  listeners:
    ...
    http_filters:
      - name: envoy.filters.http.wasm
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
          config:
            configuration: {{ .GetEnvoy.Extension.Config }} # notice use of the placeholder
            name: {{ .GetEnvoy.Extension.Name }}            # notice use of the placeholder
            vm_config:
              code: {{ .GetEnvoy.Extension.Code }}          # notice use of the placeholder
    ...
Supported placeholders
Placeholder Purpose
{{ .GetEnvoy.DefaultValue "<property>" }} Gets replaced with the default value of a given property.
{{ .GetEnvoy.Extension.Name }} Gets replaced with the extension name, e.g. "me.filters.http.my_http_filter".
{{ .GetEnvoy.Extension.Code }} Gets replaced with a Datasource representing the *.wasm file.
{{ .GetEnvoy.Extension.Config }} Gets replaced with extension config, defaults to the contents of extension.json
Supported properties

πŸ’‘ The following property names can be used in the {{ .GetEnvoy.DefaultValue "<property>" }} placeholder:

Property Gets replaced with (verbatim)
admin {"accessLogPath":"/dev/null","address":{"socketAddress":{"address":"127.0.0.1","portValue":9901}}}
admin.access_log_path "/dev/null"
admin.address {"socketAddress":{"address":"127.0.0.1","portValue":9901}}
admin.address.socket.address "127.0.0.1"
admin.address.socket.port 9901

extension.json

πŸ’‘ Example setup includes extension-specific configuration that is stored in extension.json.

Although, extensions are free to choose any format for their configuration, e.g. JSON, YAML, Protobuf, etc, it is advised to use JSON by default to keep the overall ecosystem of extensions consistent.

By default, extension.json has no contents.

Glossary

Extension workspace

Refers to the set of files generated by getenvoy extension init.

Extension descriptor

Refers to the .getenvoy/extension/extension.yaml file generated by getenvoy extension init.

Toolchain

Refers to the .getenvoy/extension/toolchains/default.yaml file generated on first use of getenvoy extension build.

Example setup

Refers to the Envoy configuration generated by getenvoy extension examples add.

Example setup represents Envoy configuration that demonstrates the extension in action on real traffic.

Example descriptor

Refers to the example.yaml file of the Example setup.

WebAssembly module

Refers to the *.wasm file generated by getenvoy extension build.