What the validating proxy checks

With --proxy-to, apigenie-server mocks nothing: every request goes to the real server, and the request and the server’s response are both checked against the document. Point a test suite, a frontend or a load test at the proxy, and the traffic becomes a contract test. This page explains what is checked and what happens with a problem; the Contract testing page of the reference lists every check.

apigenie-server pets.yaml --proxy-to http://localhost:8080 --report contract.json --fail-on errors

The examples use the pets.yaml of How the mock picks a response and a server on port 8080 that answers GET /v1/pets/1 with {"id":"1","name":"Tom"}: an id that is a string where the schema says integer.

What goes through

The server gets the request as the client sent it, without the mock’s __ query parameters and X-Apigenie-Mock-* headers, with Via and the X-Forwarded-* headers added, and with Accept-Encoding: identity so that the body of the answer can be read. A path in the target (http://localhost:8080/api) goes before the request’s path. Scenarios, examples and generated values play no part: the answer is the server’s, with the proxy’s findings in the headers.

Three groups of checks

Each problem is a violation with a check, a severity and a location. The checks fall into three groups:

The request. Is there an operation for the method and path (request.path-or-method.unknown)? Does each parameter match its schema (request.path.incompatible, request.query.incompatible, …), and is every required one there? Is the body’s media type declared, can the body be parsed, does it match its schema (request.content-type.incompatible, request.body.malformed, request.body.incompatible)? Are there credentials for a security requirement (request.authorization.missing)? Is a deprecated operation or parameter used (request.operation.deprecated)?
The response’s status and media type. Does a response cover the status, by its code, a range or default (response.status.unknown, response.status.default)? Is the media type one of the declared ones (response.content-type.incompatible)? A 5xx from the server is noted (response.status.server-error).
The response’s body and headers. Does the body match the schema of its status and media type (response.body.incompatible)? Is there a body where none is declared (response.body.unexpected)? Can it be checked at all (response.body.unvalidatable, for binary and compressed bodies)? Is every required header there and valid (response.header.incompatible)?

The answer to the example request carries one violation:

$ curl -i http://127.0.0.1:4010/v1/pets/1
HTTP/1.1 200 OK
X-Apigenie-Mock-Source: proxy
X-Apigenie-Mock-Operation: getPet
X-Apigenie-Mock-Violations: 1
X-Apigenie-Mock-Violations-Detail: [{"location":["response","body","/id"],"check":"response.body.incompatible","code":"type","severity":"error","message":"expected type integer, found string"}]

{"id":"1","name":"Tom"}

A request for GET /v1/health, which the document doesn’t describe, is forwarded too, with a request.path-or-method.unknown warning; proxy.unknown_routes: reject in the configuration answers 404 instead.

Severities

A violation is an error, a warn or an info, as the table of the reference sets it: a body that doesn’t match its schema is an error, an undeclared query parameter a warning, a deprecated operation a note. Only errors can fail a run or block traffic. contract.severity in the configuration changes the severity of a check, or turns it off with ignore:

contract:
  severity:
    response.header.undefined: warn   # off by default
    response.status.default: ignore

Report, or enforce

By default the proxy reports and changes nothing: the server’s answer goes back as it came, with the violations in X-Apigenie-Mock-Violations and X-Apigenie-Mock-Violations-Detail, in the log and in the report. Test suites send invalid requests on purpose, to see the server refuse them, and the proxy must not get in their way.

--strict (or enforce in validation.request and validation.response) turns errors into answers. A request with an error doesn’t reach the server: 400 for a body that can’t be parsed, 401 or 403 for missing or rejected credentials, 415 for an undeclared media type, 422 otherwise, each with the problems. A response with an error is replaced by a 502 that says what was wrong and what the server answered:

$ curl -i http://127.0.0.1:4010/v1/pets/1
HTTP/1.1 502 Bad Gateway
X-Apigenie-Mock-Source: mock
X-Apigenie-Mock-Operation: getPet
X-Apigenie-Mock-Upstream-Status: 200
...
  "detail": "response.body/id: expected type integer, found string",

Warnings and notes never block anything. A server that can’t be reached is a 502 of its own, one that doesn’t answer in time a 504. In fallback mode (--proxy-mode fallback) the mock answers such requests itself, as it would without a server, and says so with X-Apigenie-Mock-Upstream-Status.

How a run ends

When the server stops (Ctrl+C, or POST /__apigenie/shutdown from the test script), it prints a summary, writes the --report JSON and the --report-junit XML, and exits with 5 when the gate isn’t met: --fail-on errors (or warnings) for the violations, --min-coverage PERCENT for the part of the document the traffic exercised.

Contract: 2 requests (0 undocumented), 1 errors, 1 warnings
Coverage: 33.3% of 9 branches, 1 of 2 operations
  not tested: POST /pets (createPet)
  error GET /pets/{id}: response.body/id: expected type integer, found string (type; response.body.incompatible)
  warn GET /pets/{id}: response.body: the 404 response declares no content, but has a body of 23 bytes (unexpected; response.body.unexpected)
Contract failed: 1 error violations; coverage 33.3% is below 80.0%

Coverage counts, per operation, the operation itself, its request body (seen with a 2xx or 3xx answer) and each declared response. With a POST /pets operation added to pets.yaml, declaring a request body and the responses 201, 400 and default, the document has nine branches: four for getPet (the operation, 200, 404 and default) and five for createPet. Two requests to getPet, answered 200 and 404, cover three of them. An operation is covered, partial or not-tested.

The JSON report has the summary, the operations with their requests, the responses seen and missing and their violations, the undocumented routes, and each distinct violation with its count; its formatVersion is under the compatibility promise. The JUnit file has one test case per operation, skipped when not tested and failed with error violations, and one for the coverage threshold, which CI systems show as test results. GET /__apigenie/contract returns the report while the server runs.

The same checks elsewhere

apigenie test sends the requests itself, from .http files or generated from the document, and reports the same checks.
--record-dir writes the exchanges the server answered as stubs for the mock, and --record-har writes the session as a HAR file (Record and replay).
A GraphQL schema and the WebSocket, MQTT and HTTP channels of an AsyncAPI document are checked the same way, with checks of their own (GraphQL, AsyncAPI).