Skip to content

test: strengthen assertions and fix flaky HTTP fixtures - #273

Merged
marandaneto merged 2 commits into
mainfrom
test-audit
Sep 29, 2026
Merged

marandaneto merged 2 commits into
mainfrom
test-audit

Conversation

@marandaneto

@marandaneto marandaneto commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

💡 Motivation and Context

Several tests could pass without proving the behavior named in the test. Some counted requests instead of events, checked for no requests before flushing, or treated a dropped V1 batch as successful delivery. The flags test servers also had intermittent failures on macOS because accepted sockets inherited nonblocking mode.

This PR strengthens those assertions and fixes the local HTTP fixtures. Flag-event tests also reject capture requests that do not match the expected batches. It adds process-isolated global client lifecycle tests and checks that panicking error hooks do not prevent later hooks or worker requests. Polling tests now wait for observed requests or callbacks instead of fixed delays. Live E2E tests now check the immediate delivery result rather than only enqueueing an event.

No SDK production behavior or public API changes. No tests were removed.

💚 How did you test it?

  • The instrumented test matrix passes with default features, async capture-v1, blocking error-tracking, and blocking error-tracking with capture-v1.
  • The bare blocking configurations pass with and without capture-v1. Default and blocking V1 doctests pass.
  • The two async TCP fixture suites passed 20 consecutive runs after the repair.
  • Five controlled regressions confirmed that the strengthened tests detect incorrect variant selection, dependency matching, missing event enrichment, missing deduplication, and skipped error hooks. All mutations were restored.
  • Builds, formatting, Clippy, and the public API snapshot check pass.
  • Injecting an extra unexpected capture demonstrated a false pass in the original deduplication assertions. Catch-all assertions now detect it in all four feature configurations. After removing the injection, the full flags suite passes with 14 async V0, 14 async V1, 25 blocking V0, and 26 blocking V1 tests.
  • Autoreview passed for commit 917d28dee5b1ee69bbe5bfdb164a652396b8cf05 with no findings.
  • Coverage was measured before and after using the same feature matrix. The comparison is posted in a separate PR comment.

Live E2E configurations were compiled but not executed locally. The external compliance harness and Linux-only TLS tests were not run on this macOS host. CI reported 46/46 V0 and 111/111 V1 compliance tests passing for the initial commit.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

No release changeset is needed for these test-only changes.

@marandaneto

Copy link
Copy Markdown
Member Author

Test coverage comparison

Compared base 9649fb5c5eb6b6d7c6f3fefc6370a5f3d66ae929 with the changes in 21d2cc6cf77b2b43fc4d61265abcbc36756d967f using Rust 1.94.1 and cargo-llvm-cov 0.9.1 on macOS arm64.

Metric Before After
Production-source line coverage 90.06% (5,475 / 6,079) 91.30% (5,550 / 6,079)
SDK-source line coverage, including inline unit tests 93.22% 93.89%
SDK-source region coverage, including inline unit tests 93.43% 94.06%
SDK-source function coverage, including inline unit tests 88.74% 90.21%
Global client production-source line coverage 30.59% (26 / 85) 100% (85 / 85)

The production-source comparison adds 75 covered lines with an unchanged denominator. It excludes inline test modules and the two dedicated test-support files. Small test-only seams within production modules remain included. The other metrics are the unfiltered LLVM SDK-source report, so they include inline unit tests. These are not branch-coverage measurements.

Coverage is combined across the same four configurations before and after:

cargo llvm-cov clean --workspace
cargo llvm-cov --no-report --no-fail-fast --workspace
cargo llvm-cov --no-report --no-fail-fast --features capture-v1
cargo llvm-cov --no-report --no-fail-fast --no-default-features --features error-tracking
cargo llvm-cov --no-report --no-fail-fast --no-default-features --features error-tracking,capture-v1
cargo llvm-cov report --json --output-path coverage.json
cargo llvm-cov report --lcov --output-path coverage.lcov
Instrumented configuration Before, passed After, passed
Workspace/default 362 366
Async capture-v1 413 417
Blocking/error-tracking 336 339
Blocking/error-tracking/capture-v1 381 384

Counts overlap across configurations and are not unique test totals. Initial instrumentation exposed intermittent TCP fixture failures, which this PR fixes. The final matrix passed after the repairs. Coverage was cleaned and remeasured after restoring all controlled mutations.

Doctests were validated separately and are not included in these measurements. Live E2E, external compliance, and Linux-only TLS tests were not executed. Many repairs strengthen assertions on already-covered paths, so their value is not reflected in the coverage increase.

@greptile-apps

greptile-apps Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Retrigger

[Low risk] Test improvements and assertion strengthening.

The PR appears safe to merge; no outstanding finding or new actionable issue was identified.

Reviews (2) · Last reviewed commit: "test: reject unexpected captures in flag..."

Comment thread tests/test_evaluate_flags.rs
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

posthog-rs-v0 Compliance Report

Date: 2026-09-27 14:27:55 UTC
Duration: 15390ms

✅ All Tests Passed!

46/46 tests passed


Capture Tests

✅ 29/29 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 118ms
Format Validation.Event Has Uuid ✅ 111ms
Format Validation.Event Has Lib Properties ✅ 113ms
Format Validation.Distinct Id Is String ✅ 112ms
Format Validation.Token Is Present ✅ 110ms
Format Validation.Custom Properties Preserved ✅ 106ms
Format Validation.Event Has Timestamp ✅ 115ms
Retry Behavior.Retries On 503 ✅ 5119ms
Retry Behavior.Does Not Retry On 400 ✅ 2117ms
Retry Behavior.Does Not Retry On 401 ✅ 2117ms
Retry Behavior.Respects Retry After Header ✅ 5073ms
Retry Behavior.Implements Backoff ✅ 15076ms
Retry Behavior.Retries On 500 ✅ 5074ms
Retry Behavior.Retries On 502 ✅ 5073ms
Retry Behavior.Retries On 504 ✅ 5072ms
Retry Behavior.Max Retries Respected ✅ 15071ms
Deduplication.Generates Unique Uuids ✅ 74ms
Deduplication.Preserves Uuid On Retry ✅ 5018ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10029ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5033ms
Deduplication.No Duplicate Events In Batch ✅ 20ms
Deduplication.Different Events Have Different Uuids ✅ 18ms
Compression.Sends Gzip When Enabled ✅ 18ms
Batch Format.Uses Proper Batch Structure ✅ 39ms
Batch Format.Flush With No Events Sends Nothing ✅ 33ms
Batch Format.Multiple Events Batched Together ✅ 96ms
Error Handling.Does Not Retry On 403 ✅ 2044ms
Error Handling.Does Not Retry On 413 ✅ 2074ms
Error Handling.Retries On 408 ✅ 5053ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 41ms
Request Payload.Flags Request Uses V2 Query Param ✅ 43ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 25ms
Request Payload.Flags Request Omits Authorization Header ✅ 22ms
Request Payload.Token In Flags Body Matches Init ✅ 33ms
Request Payload.Groups Round Trip ✅ 29ms
Request Payload.Groups Default To Empty Object ✅ 32ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 32ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 29ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 30ms
Request Lifecycle.No Flags Request On Init Alone ✅ 12ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 38ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 37ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 29ms
Retry Behavior.Retries Flags On 502 ✅ 230ms
Retry Behavior.Retries Flags On 504 ✅ 224ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 33ms

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

posthog-rs-v1 Compliance Report

Date: 2026-09-27 14:28:08 UTC
Duration: 23021ms

✅ All Tests Passed!

111/111 tests passed


Capture_V1 Tests

✅ 94/94 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint ✅ 153ms
Endpoint And Method.Does Not Use Legacy Endpoints ✅ 153ms
Required Headers.Has Authorization Bearer Header ✅ 153ms
Required Headers.Has Content Type Json ✅ 153ms
Required Headers.Has Posthog Sdk Info Format ✅ 153ms
Required Headers.Has Posthog Attempt Header ✅ 152ms
Required Headers.Has Posthog Request Id ✅ 151ms
Required Headers.Has Posthog Request Timestamp ✅ 152ms
Required Headers.Has User Agent ✅ 151ms
Body Format.Body Has Created At And Batch ✅ 157ms
Body Format.No Api Key In Body ✅ 139ms
Body Format.No Sent At In Body ✅ 139ms
Event Format.Event Has Required Root Fields ✅ 139ms
Event Format.Event Uuid Is Valid ✅ 139ms
Event Format.Event Timestamp Is Rfc3339 ✅ 139ms
Event Format.Distinct Id Is String ✅ 139ms
Event Format.Distinct Id At Root Not Properties ✅ 138ms
Event Format.Custom Properties Preserved ✅ 138ms
Event Format.Set Properties Preserved ✅ 137ms
Event Format.Set Once Properties Preserved ✅ 138ms
Event Format.Groups Properties Preserved ✅ 120ms
Event Format.Sdk Generates Uuid If Not Provided ✅ 124ms
Event Format.Event Has Required Root Fields Batch ✅ 157ms
Event Format.Event Uuid Is Valid Batch ✅ 171ms
Event Format.Event Timestamp Is Rfc3339 Batch ✅ 156ms
Event Format.Distinct Id Is String Batch ✅ 155ms
Event Format.Distinct Id At Root Not Properties Batch ✅ 156ms
Event Format.Custom Properties Preserved Batch ✅ 170ms
Event Format.Set Properties Preserved Batch ✅ 155ms
Event Format.Set Once Properties Preserved Batch ✅ 166ms
Event Format.Groups Properties Preserved Batch ✅ 153ms
Event Format.Sdk Generates Uuid If Not Provided Batch ✅ 159ms
Batch Behavior.Multiple Events In Single Batch ✅ 183ms
Batch Behavior.Batch Envelope Smoke ✅ 166ms
Batch Behavior.Flush With No Events Sends Nothing ✅ 117ms
Batch Behavior.Flush At Triggers Batch ✅ 1128ms
Batch Behavior.Created At Reflects Batch Creation Time ✅ 117ms
Deduplication.Generates Unique Uuids ✅ 186ms
Deduplication.Different Events Same Content Different Uuids ✅ 126ms
Deduplication.Preserves Uuid On Retry ✅ 5125ms
Deduplication.Preserves Timestamp On Retry ✅ 5083ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5112ms
Deduplication.No Duplicate Events In Batch ✅ 138ms
Header Behavior On Retry.Attempt Header Starts At One ✅ 81ms
Header Behavior On Retry.Attempt Header Increments On Retry ✅ 10102ms
Header Behavior On Retry.Request Id Preserved On Retry ✅ 5051ms
Header Behavior On Retry.Different Requests Have Different Request Ids ✅ 2076ms
Header Behavior On Retry.Request Timestamp Changes On Retry ✅ 5054ms
Response Format Validation.Success Response Has Uuid Keyed Results ✅ 57ms
Response Format Validation.Success Response Has Ok For Each Event ✅ 35ms
Response Format Validation.Success No Retry After When All Ok ✅ 29ms
Response Format Validation.Success Retry After Present When Retry Events ✅ 30ms
Response Format Validation.Success No Retry After When Drop Only ✅ 35ms
Response Format Validation.Response Echoes Request Id ✅ 32ms
Retry Behavior.Retries On 408 ✅ 5036ms
Retry Behavior.Retries On 500 ✅ 5026ms
Retry Behavior.Retries On 503 ✅ 5026ms
Retry Behavior.Retries On 504 ✅ 5025ms
Retry Behavior.Retryable Errors Have Retry After ✅ 2026ms
Retry Behavior.Respects Retry After On Retryable Error ✅ 8027ms
Retry Behavior.Does Not Retry On 400 ✅ 2025ms
Retry Behavior.Does Not Retry On 401 ✅ 2032ms
Retry Behavior.Does Not Retry On 402 ✅ 2023ms
Retry Behavior.Does Not Retry On 413 ✅ 2028ms
Retry Behavior.Does Not Retry On 415 ✅ 2028ms
Retry Behavior.Non Retryable Errors Have No Retry After ✅ 2025ms
Retry Behavior.Implements Backoff ✅ 15031ms
Retry Behavior.Max Retries Respected ✅ 15026ms
Partial Batch Handling.Handles 200 Full Success ✅ 2027ms
Partial Batch Handling.Handles 200 With All Ok ✅ 3033ms
Partial Batch Handling.Does Not Retry Dropped Events ✅ 3031ms
Partial Batch Handling.Does Not Retry Limited Events ✅ 3030ms
Partial Batch Handling.Prunes Ok Events On Partial Retry ✅ 5027ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry ✅ 5024ms
Partial Batch Handling.Retries Only Retry Events From Partial ✅ 5027ms
Partial Batch Handling.Partial Retry Preserves Uuids ✅ 5025ms
Partial Batch Handling.Partial Retry Attempt Header Increments ✅ 5030ms
Partial Batch Handling.Partial Retry Request Id Preserved ✅ 5031ms
Partial Batch Handling.Respects Retry After On Partial ✅ 5024ms
Partial Batch Handling.Unknown Result Treated As Terminal ✅ 3025ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry ✅ 3026ms
Compression.Sends Gzip Content Encoding ✅ 22ms
Compression.No Content Encoding When Disabled ✅ 22ms
Compression.Compressed Body Is Decompressible ✅ 22ms
Error Handling.Does Not Retry On Unknown 4Xx ✅ 2024ms
Event Options.Cookieless Mode Override ✅ 23ms
Event Options.Disable Skew Correction Override ✅ 22ms
Event Options.Process Person Profile Override ✅ 23ms
Event Options.Product Tour Id Override ✅ 23ms
Event Options.Unset Options Omitted ✅ 22ms
Event Options.Options Override In Batch ✅ 22ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties ✅ 22ms
Geoip And Historical Migration.Historical Migration Set In Body ✅ 22ms
Geoip And Historical Migration.Historical Migration Absent By Default ✅ 22ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 14ms
Request Payload.Flags Request Uses V2 Query Param ✅ 14ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 15ms
Request Payload.Flags Request Omits Authorization Header ✅ 15ms
Request Payload.Token In Flags Body Matches Init ✅ 14ms
Request Payload.Groups Round Trip ✅ 15ms
Request Payload.Groups Default To Empty Object ✅ 14ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 15ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 16ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 15ms
Request Lifecycle.No Flags Request On Init Alone ✅ 14ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 21ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 17ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 14ms
Retry Behavior.Retries Flags On 502 ✅ 217ms
Retry Behavior.Retries Flags On 504 ✅ 217ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 23ms

@marandaneto
marandaneto marked this pull request as ready for review September 28, 2026 13:01
@marandaneto
marandaneto requested a review from a team as a code owner September 28, 2026 13:01
@marandaneto
marandaneto merged commit d319ce8 into main Sep 29, 2026
26 checks passed
@marandaneto
marandaneto deleted the test-audit branch September 29, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants