# Testing custom extensions

The typical development loop for custom extensions would be as follows:

1. Write the extension code
2. Start a Synapse instance with the extension loaded
3. Send requests to Synapse using a Cerbos SDK or a tool like cURL to exercise the extension

Synapse includes a testing framework to simplify the above process. It enables test-driven development and automated testing of custom extensions under various conditions. Tests are written as Starlark scripts but they can be used to test any kind of WASM or Starlark extension. For each test suite, the test runner starts a fresh instance of Synapse using the configuration defined in the suite and executes the test functions against that instance. There is a rich library of utility functions available to craft and send requests to exercise the extensions under test.

## A simple test suite

The test runner expects test suites to be defined in files that have names ending with `_test.star` such as `my_awesome_test_suite_test.star`. Test cases must be functions that begin with the prefix `test_` and accept a single `context` parameter. Any function that doesn’t start with `test_` will be ignored by the test runner so you are free to define other functions that don’t begin with `test_` to help you organize your test code.

The following example shows a simple test suite with a test case named `always_pass` that always passes and another one called `always_fail` that always fails.

```python
# This test case always passes
def test_always_pass(context):
    return testing.ok("I always pass")

# This test case always fails
def test_always_fail(context):
    return testing.fail("I always fail")

# This function is ignored by the test runner because it doesn't start with 'test_'
def ignored():
    pass
```

Save the contents of the above as `example_test.star` and execute the test suite.

```console
$ docker run 
    -v $(pwd):/tests:ro 
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest 
    test /tests
```

It should produce output similar to the following:

```
== FAIL:example (/tests/example_test.star)
   Passed=1 Failed=1 Skipped=0
   --- FAIL:always_fail | I always fail
   --- PASS:always_pass | I always pass
```

The above example is a bare minimum test suite to illustrate how the test runner works. Read on to learn how to write more useful tests that actually exercise your extensions.

## Defining test suite parameters

The test runner provisions a fresh Synapse instance for each test suite. By default it is a basic server with no extensions and an in-process PDP configured to read policies from a directory named `policies` in the current working directory. You can provide your own Synapse configuration to use in the test suite by defining a `test_suite` object.

```python
test_suite = struct(
   synapse_config = testing.load_synapse_config("synapse.yaml"),
)
```

The above example instructs the test runner to start a Synapse instance using the configuration read from the file named `synapse.yaml`. It should be a valid configuration file as documented in [Configuration](https://docs.cerbos.dev/synapse/latest/configuration/reference).

It is also possible to provide Synapse configuration inline as a Starlark dictionary.

```python
test_suite = struct(
   synapse_config = {
        "pdp": {
            "inProcess": {
                "storage": {
                    "driver": "disk",
                    "disk": {
                        "directory": "${SYNAPSE_ROOT}/policies"
                    }
                }
            }
        }
   }
)
```

|     |     |
| --- | --- |
|  | The test runner sets the `SYNAPSE_ROOT` environment variable to the directory of the test file. You can use this in the configuration file to reference other files that are stored relative to the test file. |

By default the test suite name is derived from the file name. You can provide a more descriptive name using the `name` attribute.

```python
test_suite = struct(
   name = "Tests for extension foo",
   synapse_config = testing.load_synapse_config("synapse.yaml")
)
```

Set the `skip` attribute to make the test runner skip executing the tests defined in this test suite.

```python
test_suite = struct(
   name = "Tests for extension foo",
   synapse_config = testing.load_synapse_config("synapse.yaml"),
   skip = True
)
```

If an attribute named `test_cases` is defined, it specifies the test functions that belong to this test suite. It overrides the implicit test discovery mechanism and the test runner will only run the functions that are listed in the `test_cases` attribute.

```python

def always_fail(context):
    return testing.fail("I always fail")

test_suite = struct(
   name = "Tests for extension foo",
   synapse_config = testing.load_synapse_config("synapse.yaml"),
   test_cases = {
      "always_pass": lambda context: testing.ok("I always pss"), # 1
      "always_fail": always_fail # 2
   }
)
```

|     |     |
| --- | --- |
| **1** | A test function defined inline as a lambda |
| **2** | Reference to a test function defined elsewhere in the file |

|     |     |
| --- | --- |
|  | In the example above, the `always_fail` function needs to be declared first before it’s referenced by the `test_suite` object because Starlark is a dialect of Python that always evaluates top to bottom. Things must be declared before they are referenced. |

## The `testing` module

Every test suite has access to the built-in `testing` module which provides utilty functions for writing Synapse tests.

| Name | Description |
| --- | --- |
| `testing.assert(expression)` | Assert that the expression evaluates to true |
| `testing.load_synapse_config(path)` | Load a Synapse configuration file from the given path |
| `testing.load_testdata(path)` | Load a test data file from the give path (see [reusing test data](https://docs.cerbos.dev/synapse/latest/extensions/testing#testdata) section for more details) |
| `testing.ok(message)` | Return this to explicitly mark the test as passed |
| `testing.fail(message)` | Return this to explicitly mark the test as failed |
| `testing.skip(message)` | Return this to explicitly mark the test as skipped |

## Setting the status for a test

The return value of the test function determines its status. If the function returns nothing, a `True` value or `testing.ok()`, it’s considered to have passed. Triggering a runtime error, returning `False` or `testing.fail()` marks the test as failed. Returning `testing.skip()` marks the test as skipped.

## The test case context

The `context` object passed to each test case function contains helper methods to make requests to the Synapse instance of that test suite.

```python
test_suite = struct(
   name = "Illustrate how to exercise different extension types",
   synapse_config = testing.load_synapse_config("synapse.yaml"),
)

# Send a Cerbos CheckResources request to exercise configured proxy extensions
def test_check_resources(context):
    request = struct(
        request_id = "test",
        principal = struct(id = "john", roles = ["employee"]),
        resources = [struct(
            actions = ["view"],
            resource = struct(
                kind = "invoice",
                id = "XX125",
                attr = { "owner": "john", "department": "IT", "geography": "GB"  }
            )
        )]
    )

have = context.check_resources(request)
    return testing.assert(have.results[0].actions["view"] == "EFFECT_ALLOW")

# Send a Cerbos PlanResources request to exercise configured proxy extensions
def test_plan_resources(context):
    request = struct(
        request_id = "test",
        principal = struct(id = "john", roles = ["employee"]),
        actions = ["view"],
        resource = struct(kind = "invoice")
    )

have = context.plan_resources(request)
    return testing.assert(set(have.actions) == set(["view"]) and have.filter.kind == "KIND_CONDITIONAL")

# Send an AuthZEN access evaluation request to exercise configured proxy extensions
def test_access_evaluation(context):
    request = struct(
        subject = struct(
            type = "user",
            id = "john",
            properties = {"cerbos.roles": ["employee"]}
        ),
        resource = struct(
            type = "invoice",
            id = "XX125",
            properties =  { "owner": "john", "department": "IT", "geography": "GB"  }
        ),
        action = struct(name = "view")
    )

have = context.access_evaluation(request)
    return testing.assert(have.decision)

# Send an AuthZEN access evaluation batch request to exercise configured proxy extensions
def test_access_evaluation_batch(context):
    request = struct(
        subject = struct(
            type = "user",
            id = "john",
            properties = {"cerbos.roles": ["employee"]}
        ),
        resource = struct(
            type = "invoice",
            id = "XX125",
            properties =  { "owner": "john", "department": "IT", "geography": "GB"  }
        ),
        evaluations = [struct(action = struct(name = "view"))]
    )

have = context.access_evaluation_batch(request)
    return testing.assert(have.evaluations[0].decision)

# Send an extenal authz request to the Envoy extension
def test_envoy_check(context):
    request = struct(
        attributes = struct(
            source = struct(
                principal = "spiffe://cerbos.dev/cerbie"
            ),
            request = struct(
                http = struct(
                   id = "foo",
                   method = "GET",
                   path = "/foo?op=check_response"
                ),
            ),
        ),
    )

have = context.envoy_check(request)
    return testing.assert(have.status.code == 0)

# Send an HTTP request to a configured route extension
def test_route_extension(context):
   have = context.http_get("/ext/foo", {"op": "http_response"})
   return testing.assert(have.status_code == 200 and have.body() == "Welcome")
```

## Reusing test data

You can reuse common requests and responses by defining them in a test data file. They can be loaded into the test suite using the `testing.load_testdata` function. The test data file must be a YAML or JSON file in the following format:

```yaml
testData: 
  req_john: 
    checkResourcesRequest: 
      requestId: "john"
      principal:
        id: "john"
        roles: ["employee"]
      resources:
        - resource:
            id: "XX125"
            kind: "invoice"
            attr:
              owner: "john"
              department: "Music"
              geography: "GB"
          actions: ["view"]

resp_john:
    checkResourcesResponse:
      requestId: "john"
      results:
        - resource:
            id: "XX125"
            kind: "invoice"
          actions:
            view: EFFECT_ALLOW
```

|     |     |
| --- | --- |
| **1** | Required. The file’s root object must be named `testData`. |
| **2** | Identifier for the data object. This is the name you use to reference different test data objects. |
| **3** | Type of the data. See below for acceptable values. |

The following types of data can be included in a test data file.

| Type | Schema |
| --- | --- |
| `authzenEvaluationBatchRequest` | [authzen.authorization.v1.AccessEvaluationBatchRequest](https://buf.build/cerbos/cerbos-api/docs/main%3Aauthzen.authorization.v1#authzen.authorization.v1.AccessEvaluationBatchRequest) |
| `authzenEvaluationBatchResponse` | [authzen.authorization.v1.AccessEvaluationBatchResponse](https://buf.build/cerbos/cerbos-api/docs/main%3Aauthzen.authorization.v1#authzen.authorization.v1.AccessEvaluationBatchResponse) |
| `authzenEvaluationRequest` | [authzen.authorization.v1.AccessEvaluationRequest](https://buf.build/cerbos/cerbos-api/docs/main%3Aauthzen.authorization.v1#authzen.authorization.v1.AccessEvaluationRequest) |
| `authzenEvalationResponse` | [authzen.authorization.v1.AccessEvaluationResponse](https://buf.build/cerbos/cerbos-api/docs/main%3Aauthzen.authorization.v1#authzen.authorization.v1.AccessEvaluationResponse) |
| `checkResourcesRequest` | [cerbos.request.v1.CheckResourcesRequest](https://buf.build/cerbos/cerbos-api/docs/main%3Acerbos.request.v1#cerbos.request.v1.CheckResourcesRequest) |
| `checkResourcesResponse` | [cerbos.response.v1.CheckResourcesResponse](https://buf.build/cerbos/cerbos-api/docs/main%3Acerbos.response.v1#cerbos.response.v1.CheckResourcesResponse) |
| `envoyCheckRequest` | [envoy.service.auth.v3.CheckRequest](https://buf.build/envoyproxy/envoy/docs/main:envoy.service.auth.v3#envoy.service.auth.v3.CheckRequest) |
| `envoyCheckResponse` | [envoy.service.auth.v3.CheckResponse](https://buf.build/envoyproxy/envoy/docs/main%3Aenvoy.service.auth.v3#envoy.service.auth.v3.CheckResponse) |
| `planResourcesRequest` | [cerbos.request.v1.PlanResourcesRequest](https://buf.build/cerbos/cerbos-api/docs/main%3Acerbos.request.v1#cerbos.request.v1.PlanResourcesRequest) |
| `planResourcesResponse` | [cerbos.response.v1.PlanResourcesResponse](https://buf.build/cerbos/cerbos-api/docs/main%3Acerbos.response.v1#cerbos.response.v1.PlanResourcesResponse) |

```python
example_test.star

testdata = testing.load_testdata("check-resources-test-data.yaml")

def test_external_test_data(context):
    request = testdata["req_john"] 
    have = context.check_resources(request)
    want = testdata["resp_john"]
    return testing.assert(have.results[0].actions["view"] == want.results[0].actions["view"])
```

## Parameterized tests

To avoid repetition, you can generate test cases within the test suite itself. The following example illustrates generating test cases using a test data file. The test data in this case is organized into request and response pairs (IDs prefixed with `req_` and `resp_`). The script iterates through all the requests and generates test cases for each that checks whether the actual response matches the expected response.

```python
def build_test_case(request, expected):
    def test_case(context):
        actual = context.check_resources(request)
        for (i, expected_result) in enumerate(expected.results):
            for (action, expected_effect) in dict(expected_result.actions).items():
                if actual.results[i].actions[action] != expected_effect:
                    return testing.fail("Action {} of result {} should be {} but is {}".format(action, i, expected_effect, actual.results[i].actions[action]))

return test_case

def build_test_case_list():
    testdata = testing.load_testdata("check-resources-test-data.yaml")
    test_cases = {}
    for (name, data) in testdata.items():
        if name.startswith("req_"):
            test_name = name.removeprefix("req_")
            expected_resp = testdata.get("resp_" + test_name)
            if expected_resp:
                test_cases[test_name] = build_test_case(data, expected_resp)

return test_cases

test_suite = struct(
    name = "CheckResources tests",
    synapse_config = testing.load_synapse_config("synapse.yaml"),
    test_cases = build_test_case_list() 
)
```

## Loading modules

You can `load` any of the [supported modules](https://docs.cerbos.dev/synapse/latest/extensions/starlark-development#loadable-modules) for use in tests.

```python
load("re", "re")

def test_regex(context):
    return testing.assert(re.match("f.*", "foo"))
```

## Running a subset of tests

Multiple directories and files can be passed as arguments to the `test` command. When an argument is a directory, it’s recursively searched to discover all files that have the `_test.star` suffix. If you want to run just a single test suite, pass the full path to the file. If you want to run only test cases that match a particular pattern, use the `--match` argument to pass a regular expression.

```console
# Discover all the test suites under /tests and run them
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test /tests

# Run only extensionx_test.star and extensiony_test.star suites
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test /tests/extensionx/extensionx_test.star /tests/extensiony/extensiony_test.star

# Run only test cases whose names begin with "enrich"
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test --match='^enrich' /tests
```

If you need to further process the test results (using a script, for instance) pass `--output=jsonl` to make the test runner produce output in JSONLines format.

## Debugging failures

Pass `--verbose` to see the logs from Synapse instance and any output produced by the test cases through `print()`.

```console
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test --verbose /tests
```

You can use the [Starlark REPL](https://docs.cerbos.dev/synapse/latest/extensions/starlark-development#repl) to help debug issues with your test scripts. Try out tricky expressions in the REPL to figure out how they behave or load the test script in and invoke its functions by hand to test them.

```console
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    starlark repl /tests/fail_test.star
```