Skip to content

Improve /events filters and add documentation #643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nuclearcat wants to merge 1 commit into
base: main
Choose a base branch
from
+208 −12
Open

Improve /events filters and add documentation #643

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 30 additions & 12 deletions api/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
from pydantic import BaseModel
from jose import jwt
from jose.exceptions import JWTError
from kernelci.api.models import (

Check failure on line 48 in api/main.py

View workflow job for this annotation

GitHub Actions / Lint 

Unable to import 'kernelci.api.models'
Node,
Hierarchy,
PublishEvent,
Expand Down Expand Up @@ -777,15 +777,29 @@
Returns (recursive, processed_query_params).
Raises HTTPException on validation errors.
"""
# Simple filters: param_name -> query_field
simple_filters = {
'kind': 'data.kind',
'state': 'data.state',
'result': 'data.result',
'op': 'data.op',
'name': 'data.name',
'group': 'data.group',
'owner': 'data.owner',
'channel': 'channel',
}

recursive = query_params.pop('recursive', None)
limit = query_params.pop('limit', None)
kind = query_params.pop('kind', None)
state = query_params.pop('state', None)
result = query_params.pop('result', None)
from_ts = query_params.pop('from', None)
node_id = query_params.pop('node_id', None)
event_id = query_params.pop('id', None)
event_ids = query_params.pop('ids', None)
path = query_params.pop('path', None)

# Apply simple filters
for param, field in simple_filters.items():
value = query_params.pop(param, None)
if value:
query_params[field] = value

if node_id:
if 'data.id' in query_params:
Expand All @@ -795,18 +809,16 @@
)
query_params['data.id'] = node_id

event_id = query_params.pop('id', None)
event_ids = query_params.pop('ids', None)
_parse_event_id_filter(query_params, event_id, event_ids)

if from_ts:
if isinstance(from_ts, str):
from_ts = datetime.fromisoformat(from_ts)
query_params['timestamp'] = {'$gt': from_ts}
if kind:
query_params['data.kind'] = kind
if state:
query_params['data.state'] = state
if result:
query_params['data.result'] = result
if path:
query_params['data.path'] = {'$regex': path}
if limit:
query_params['limit'] = int(limit)

Expand All @@ -828,10 +840,16 @@
Get all the matching events otherwise.
Query parameters can be used to filter the events:
- limit: Number of events to return
- from: Start timestamp (unix epoch) to filter events
- from: Start timestamp (unix epoch or ISO format) to filter events
- kind: Event kind to filter events
- state: Event state to filter events
- result: Event result to filter events
- op: Operation type ('created', 'updated')
- name: Node name to filter events
- path: Node path to filter events (regex pattern)
- group: Node group to filter events
- owner: Node owner to filter events
- channel: Pub/sub channel to filter events
- id / ids: Event document id(s) to filter events
- node_id: Node id to filter events (alias for data.id)
- recursive: Retrieve node together with event
Expand Down
178 changes: 178 additions & 0 deletions doc/api-details.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -941,3 +941,181 @@ kernelci-api | INFO: 127.0.0.1:35810 - "POST /publish/abc HTTP/1.1" 200 OK
kernelci-api | INFO: 127.0.0.1:35754 - "GET /listen/abc HTTP/1.1" 200 OK
kernelci-api | INFO: 127.0.0.1:36744 - "POST /unsubscribe/abc HTTP/1.1" 200 OK
```


## Events

The `/events` endpoint provides access to the event history stored in MongoDB.
Events are generated when nodes are created or updated, and are stored for a
configurable retention period (default 7 days).

### Event Structure

Each event contains the following fields:

| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Unique event document ID |
| `timestamp` | datetime | When the event was created |
| `sequence_id` | integer | Sequential ID for ordering (used by pub/sub) |
| `channel` | string | Pub/sub channel name (typically "node") |
| `owner` | string | Username of the event publisher |
| `data` | object | Event payload (see below) |

The `data` object contains node information:

| Field | Type | Description |
|-------|------|-------------|
| `op` | string | Operation type: `created` or `updated` |
| `id` | string | Node ID |
| `kind` | string | Node kind (e.g., `checkout`, `kbuild`, `job`, `test`) |
| `name` | string | Node name |
| `path` | array | Node path hierarchy |
| `group` | string | Node group |
| `state` | string | Node state: `running`, `available`, `closing`, `done` |
| `result` | string | Node result: `pass`, `fail`, `skip`, `incomplete`, or `null` |
| `owner` | string | Node owner (username) |
| `data` | object | Node-specific data |
| `is_hierarchy` | boolean | Whether this is a hierarchy update |

### Query Parameters

The `/events` endpoint supports the following query parameters:

| Parameter | Type | Description | Example |
|-----------|------|-------------|---------|
| `limit` | integer | Maximum number of events to return | `limit=100` |
| `from` | datetime | Return events after this timestamp (ISO format or Unix epoch) | `from=2025-01-01T00:00:00` |
| `kind` | string | Filter by node kind | `kind=job` |
| `state` | string | Filter by node state | `state=done` |
| `result` | string | Filter by node result | `result=pass` |
| `op` | string | Filter by operation type | `op=created` |
| `name` | string | Filter by node name (exact match) | `name=baseline-x86` |
| `path` | string | Filter by node path (regex pattern) | `path=.*mainline.*` |
| `group` | string | Filter by node group (exact match) | `group=kunit-x86_64` |
| `owner` | string | Filter by node owner (exact match) | `owner=admin` |
| `channel` | string | Filter by pub/sub channel | `channel=node` |
| `id` | string | Filter by event document ID | `id=507f1f77bcf86cd799439011` |
| `ids` | string | Filter by multiple event IDs (comma-separated) | `ids=id1,id2,id3` |
| `node_id` | string | Filter by node ID (alias for `data.id`) | `node_id=507f1f77bcf86cd799439011` |
| `recursive` | boolean | Include full node data with each event | `recursive=true` |

> **Note**: When using `recursive=true`, the `limit` parameter is required and must be <= 1000.

### Examples

**Get all events (limited to default pagination):**
```
$ curl http://localhost:8001/latest/events
```

**Get events for completed jobs with passing results:**
```
$ curl 'http://localhost:8001/latest/events?kind=job&state=done&result=pass'
```

**Get recently created events (last hour):**
```
$ curl 'http://localhost:8001/latest/events?op=created&from=2025-01-10T12:00:00'
```

**Get events for a specific node path pattern (regex):**
```
$ curl 'http://localhost:8001/latest/events?path=.*linux-next.*&limit=50'
```

**Get events for a specific group:**
```
$ curl 'http://localhost:8001/latest/events?group=kunit-x86_64&state=done'
```

**Get events by owner:**
```
$ curl 'http://localhost:8001/latest/events?owner=admin&kind=checkout'
```

**Get events with full node data:**
```
$ curl 'http://localhost:8001/latest/events?state=done&result=fail&recursive=true&limit=10'
```

**Get events for a specific node ID:**
```
$ curl 'http://localhost:8001/latest/events?node_id=65a1355ee98651d0fe81e412'
```

**Combine multiple filters:**
```
$ curl 'http://localhost:8001/latest/events?kind=test&state=done&result=fail&group=kselftest&from=2025-01-01&limit=100'
```

### Sample Response

```json
[
{
"id": "65a1355ee98651d0fe81e500",
"timestamp": "2025-01-12T08:30:00.000000",
"sequence_id": 12345,
"channel": "node",
"owner": "admin",
"data": {
"op": "updated",
"id": "65a1355ee98651d0fe81e412",
"kind": "test",
"name": "kselftest-cpufreq",
"path": ["checkout", "kbuild", "test", "kselftest-cpufreq"],
"group": "kselftest",
"state": "done",
"result": "pass",
"owner": "admin",
"data": {
"kernel_revision": {
"tree": "mainline",
"branch": "master",
"commit": "abc123..."
}
},
"is_hierarchy": false
}
}
]
```

### Response with recursive=true

When `recursive=true` is specified, each event includes a `node` field with the
full node object:

```json
[
{
"id": "65a1355ee98651d0fe81e500",
"timestamp": "2025-01-12T08:30:00.000000",
"data": {
"op": "updated",
"id": "65a1355ee98651d0fe81e412",
"kind": "test",
"name": "kselftest-cpufreq",
"state": "done",
"result": "pass"
},
"node": {
"id": "65a1355ee98651d0fe81e412",
"kind": "test",
"name": "kselftest-cpufreq",
"path": ["checkout", "kbuild", "test", "kselftest-cpufreq"],
"group": "kselftest",
"parent": "65a1355ee98651d0fe81e400",
"state": "done",
"result": "pass",
"artifacts": {...},
"data": {...},
"created": "2025-01-12T08:00:00.000000",
"updated": "2025-01-12T08:30:00.000000",
"owner": "admin",
"user_groups": []
}
}
]
```