Problem
export_user_data (app/services/gdpr.py) caps the audit history it exports:
audit_entries = (
db.query(AuditLogORM)
.filter((AuditLogORM.email == user.email) | (AuditLogORM.actor_id == user.id))
.order_by(AuditLogORM.created_at.desc())
.limit(1000)
.all()
)The response reports audit_log_count: len(audit_logs) — the count of the truncated list — with no indication that more rows exist.
Consequences:
- The export is legally incomplete without signaling it: a data-subject request is supposed to return all personal data; users with more than 1000 audit events receive a subset and have no way to know, so the GDPR artifact is non-compliant and misleading.
- The count field reinforces the false impression:
audit_log_count: 1000 reads as the full count, not "1000 of N".
- The 1000-entry bound exists only to keep the synchronous export fast (the design tracked separately); it is a performance shortcut applied to a legal obligation.
Root cause
The export's completeness was traded away for the "< 30 s" performance goal without a completion signal or pagination.
Why this is architecturally hard
- Completeness requires either iterating all matching rows (with the async/storage design tracked separately to bound memory) or clearly signaling truncation (
truncated: true, total_matching: N, a continuation mechanism) — a contract decision for the response schema.
- The query itself can be slow at volume (two ORed indexed columns); returning the true total needs a
count() alongside, which doubles the query cost unless it is a window function.
- The response schema is in the OpenAPI snapshot and consumed by the frontend; adding truncation fields is a coordinated change.
Proposed design
Report the true matching total and a truncated flag (or paginate the export), never silently limiting the legal artifact; add a test with 1001+ matching entries asserting the response signals the omission.
Acceptance criteria
Service
Tests
Out of scope
The synchronous in-memory tarball (tracked separately) and erasure path.
Getting started
pytest tests/test_gdpr.py -q
make typecheck
Good first files to read: app/services/gdpr.py, app/api/v1/endpoints/auth.py.
Problem
export_user_data(app/services/gdpr.py) caps the audit history it exports:The response reports
audit_log_count: len(audit_logs)— the count of the truncated list — with no indication that more rows exist.Consequences:
audit_log_count: 1000reads as the full count, not "1000 of N".Root cause
The export's completeness was traded away for the "< 30 s" performance goal without a completion signal or pagination.
Why this is architecturally hard
truncated: true,total_matching: N, a continuation mechanism) — a contract decision for the response schema.count()alongside, which doubles the query cost unless it is a window function.Proposed design
Report the true matching total and a
truncatedflag (or paginate the export), never silently limiting the legal artifact; add a test with 1001+ matching entries asserting the response signals the omission.Acceptance criteria
Service
Tests
Out of scope
The synchronous in-memory tarball (tracked separately) and erasure path.
Getting started
Good first files to read:
app/services/gdpr.py,app/api/v1/endpoints/auth.py.