Surf OOAPI Gateway

Status

Master branch status:

Setting up development environment

Install nodejs for your system.

Fetch express-gateway-lite submodule:

git submodule update --init --recursive

Install other dependencies:

npm install

Set SECRETS_KEY_FILE environment variable to config/test-secret.txt.

Running tests

Functional tests

Run all tests with:

npm test

Run all tests and see docker output:

DEBUG='testcontainers*' npm test

Run all tests and see gateway logging output:

MOCHA_LOG_GW_TO_CONSOLE=true npm test

Run all but the integration tests with:

MOCHA_SKIP=integration npm test

For development and tests the default gateway configuration points at two test backends. These can be started locally with:

npm run test-backend &
npm run other-test-backend &

Performance/stress tests

The performance test scripts use apachebench, so make sure ab is installed on your PATH.

On MacOs it may already be available at /usr/sbin/ab

On debian:

apt-get install apache2-utils

To run the tests do

./scripts/perf-test.js --verbose=2

This will attempt to run 1000 requests, first sequentially, and then with increasing concurrency (up to 250 concurrent requests).

So see the available options do

./scripts/perf-test.js --help

Security

See doc/security.org on the strategies we use to keep the gateway secure.

Client authentication

Use the ./bin/credentials command line tool to create client credentials in the correct encoding.

./bin/credentials create myapp

Will output something like

  Config entry:
	myapp: {
         "passwordSalt":"f2c6d54c0aa39cde114702920b84a753",
         "passwordHash":"eaaeaf91f8e5df9daa88c6980d057eb980757632ebea33b4c2060fef33a31ba2"
       }

  Basic Auth token:
	myapp:64be12c92c1d49ba12d5279e3b444705

You will then need to update the ./config/gateway.config.yml file and put the given config entry in the gatekeeper apps section to be used with basic authentication. In the following example myapp is the username part and 0123456789abcdef0123456789abcdef is the password.

myapp:0123456789abcdef0123456789abcdef

Please note: only a hash of the password is stored, the password itself is not stored.

Client authorization

<<client-auth>>

To determine which app can access which paths on which endpoints a collection of ACLs is configured on the gatekeeper policy in ./config/gateway.config.yml.

Per app the accessible endpoints are listed and the paths on that endpoint. The paths are formatted as path expressions like /user/:name where :name is a variable path component.

In the following example, the app named fred has access to the endpoints wilma and betty. On wilma it can access / and any “dinner” resource like /dinner/tonight or /dinner/tomorrow but not /dinner. It can also access /visits on the betty endpoint but nothing else.

- gatekeeper:
    - action:
        acls:
          - app: fred
            endpoints:
            - endpoint: wilma
              paths: ['/', '/dinner/:date']
            - endpoint: betty
              paths: ['/visits']

The endpoint(s) an application tries to access is derived from the X-Route header. The gatekeeper policy expects this header to have a directive which starts with endpoint= followed by a comma separated list of endpoint identifiers. The endpoint identifiers may only contain alphanumeric characters.

In the following example access to both wilma and betty is requested.

X-Route: endpoint=wilma,betty

Only the endpoint directive is supported at this point, any value for the X-Route header not starting with endpoint= is ignored.

Server authentication

The proxyOptions described below should be encoded using the 192 bit hexadecimal key referred by the SECRETS_KEY_FILE environment variable. Use ./bin/encode-proxy-options encode proxyOptions to proxyOptionsEncoded using the key in SECRETS_KEY_FILE.

Basic authentication

Service endpoints can be secured using basic authentication by adding proxyOptions.auth options. Here’s an example:

serviceEndpoints:
  BoulderCollege:
    url: https://boulder-college.co/ooapi/
    proxyOptions:
      auth: fred:wilma

OAuth2 Client Credentials

Service endpoints can be secured using the OAuth2 client credentials grant type [fn:oauth2-ccg:See also RFC 6749 section 4.4]. Here’s an example:

serviceEndpoints:
  BoulderCollege:
    url: https://boulder-college.co/ooapi/
    proxyOptions:
      oauth2:
        clientCredentials:
          tokenEndpoint:
            url: https://college-oauth.co/token
            params:
              grant_type: client_credentials
              client_id: fred
              client_secret: wilma

Notes:

• params are the exact request parameters for the token endpoint, this is also the location to add scope when needed
• only passing credentials through params is supported at this time although RFC mentions basic authentication[fn:oauth2-ccg-atr:See also RFC 6749 section 4.4.2].

Special API key headers

Service endpoints depending on special API key headers to authorize use can be configured through proxyOptions.headers. In the following example a “Authorization” is expected with a bearer token:

serviceEndpoints:
  BoulderCollege:
    url: https://boulder-college.co/ooapi/
    proxyOptions:
      headers:
        Authorization: "Bearer <myverysecrettoken>"

Note: any header can be added here.

Configuration

Logging

Request logging according to the GELF format is implemented using the lifecycle-logger policy, which logs to STDOUT.

- lifecycle-logger:
  - action:

The following properties are logged for incoming requests:

• short_message: the request method
• trace_id: the requestId generated by Express Gateway
• client: the app id
• http_status: the HTTP status code of the response
• url: the path of the incoming request
• time_ms: the number of milliseconds it took to respond

Logging of outgoing requests to the backends is built in to the aggregation policy. Outgoing requests are also logged using GELF and have the following properties:

• short_message: the request method
• trace_id: the requestId of the corresponding incoming request
• client: always ‘PROXY’
• http_status: the HTTP status code of the response
• url: the full url of the outgoing request
• time_ms: the number of milliseconds it took to respond

Incoming and outgoing requests can be correlated using the trace_id.

Log forwarding

In production logs are forwarded to a Greylog server. In development you can test this setup using the services in ./dev/observability/docker-compose.yml. See also ./dev/observability/README.md and ./dev/docker-compose-with-logging-and-redis.yml

Endpoint timeouts

Service endpoints can be configured to have a strict timeout policy by adding a proxyOptions.proxyTime option in milliseconds. This is the maximum time allow for an endpoint to respond. Here’s an example:

serviceEndpoints:
  BoulderCollege:
    url: https://boulder-college.co/ooapi/
    proxyOptions:
      proxyTimeout: 10000

TLS

Setting server certificates

To serve https requests, you need to specify your private key and the signed certificate as follows

https:
  port: 4444
  tls:
    default: # replace with real certificate in prod environment
      key: "config/testServer.key"
      cert: "config/testServer.crt"

Generating self-signed certificates

The integration tests allow self-signed certificates, which you can generate as follows:

# create root certificate authority for signing our own certs
cd config
openssl genrsa -out testRootCA.key 2048
openssl req -x509 -new -nodes -key testRootCA.key -sha256 -days 1024 -out testRootCA.pem

# create server certificate
openssl req -nodes -newkey rsa:2048 -keyout testServer.key -out testServer.csr
openssl x509 -req -days 365 -in testServer.csr -CA testRootCA.pem -CAkey testRootCA.key -set_serial 01 -out testServer.crt

Request/response validation

Requests and responses can be validated against the OOAPI specification using the openapi-validator policy.

- openapi-validator:
   - action:
       apiSpec: 'ooapiv4.json'
       validateRequests: true
       validateResponses: true

When validateRequests is true, all incoming requests are validated.

When validateResponses is true, responses are validated when the request has an X-Validate-Response: true header.

OOAPI V4 & V5 configuration & validation

There are example configurations for handling and validating OOAPI v4 and v5 at config/gateway.config.yml.v4 and config/gateway.config.yml.v5. These include the correct set of endpoint paths for each version, and refer to the API specifications at ooapiv4.json and ooapiv5.json.

The ooapiv5 json can be regenerated from the specification repository (included as a submodule in ooapi-specification) by running:

make ooapiv5.json

Which will generate a version of ooapi specification that excludes the response schemas since the full v5 specification is incompatible with the validation library used.

You will need to have jq installed. On MacOS, jq is available with brew:

brew install jq

Aggregation

The aggregation policy will send requests to a number of endpoints in parallel and return an envelope containing the individual responses.

The endpoints are determined by the the X-Route header, which contains a list of serviceEndpoint identifiers. If no X-Route header is provided, all enabled endpoints in the client’s ACL are used.

X-Route: endpoint=tue,wur

See also Client Authorization.

Aggregation, responses, and response validation

When responses from multiple backends are aggregated, they are wrapped in an envelope.

Aggregation has the following config options

noEnvelopIfAnyHeaders

- aggregation:
    - action:
        noEnvelopIfAnyHeaders:
          'X-Validate-Response': 'true'
          'X-Envelope-Response': 'false'

Since aggregated responses are never valid against the OOAPI spec, the gateway will not aggregate when X-Validate-Response: true is specified. In this case, the request must specify an X-Route header with exactly one backend, or a BAD REQUEST response is returned.

Applications that request a single endpoint’s data according in OOAPI format can also provide an X-Envelope-Response: false header, which disables the envelope, passing the endpoints response body as-is. As with the X-Validate-Response header, the request must specify an X-Route header with exactly one backend, or a BAD REQUEST response is returned.

keepRequestHeaders

- aggregation:
    - action:
        keepRequestHeaders:
          - 'accept'
          - 'accept-language'

When keepRequestHeaders is specified it lists all headers from the client that will be forwarded to the backends.

If keepRequestHeaders is not specified all headers will be forwarded.

keepResponseHeaders

- aggregation:
    - action:
        keepResponseHeaders:
          - 'content-type'
          - 'content-length'

When keepResponseHeaders is specified it lists all headers from the endpoints that will be returned to the backends.

If keepResponseHeaders is not specified all headers will be returned.

proxyOptions

Allow setting global proxyOptions (can be overwritten per backend).

- aggregation:
    - action:
        proxyOptions:
          proxyTimeout: 10000

The above will set a global timeout for requests going to backends of 10 seconds.

Building deployable images

The repository includes a Dockerfile that can be used to build a deployable docker image, including the configuration provided in the ./config directory.

Ensure Docker is installed and do the usual:

docker build .

to build the image.

Use the following environment variables to setup the Redis username and password in config/system.config.yml:

• REDIS_USERNAME
• REDIS_PASSWORD

Note that not using authorization for Redis is also possible by editing config/system.config.yml to delete the db.redis.username and db.redis.password properties before building the docker image.

Location of system.config and gateway.config files and docker mounts

The docker container does not contain a gateway configuration file.

By default, the gateway loads system.config.yml and gateway.config.yml from the ./config directory. The ./config directory contains additional model files that must be present.

To run the gateway in docker with a mutable configuration, it’s recommended to read-write mount a directory containing the gateway configuration files and to set the EG_SYSTEM_CONFIG_PATH and EG_GATEWAY_CONFIG_PATH environment variables to point to the files to use in that directory.

Setting up development environment

The OOAPI Gateway runs on NodeJS. For an ergonomic development environment you need Docker and NodeJS + NPM.

Installing node dependencies

Install the JS dependencies in the ./node_modules local directory. You do not need to install any modules globally.

npm install

Running the gateway in development

npm start

Running tests

npm test

Reporting vulnerabilities

If you have found a vulnerability in the code, we would like to hear about it so that we can take appropriate measures as quickly as possible. We are keen to cooperate with you to protect users and systems better. See https://www.surf.nl/.well-known/security.txt for information on how to report vulnerabilities responsibly.

License

Copyright (C) 2020 SURFnet B.V.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.