Repository navigation
[fix] Store background task timestamps as naive UTC - #5142
Open
theluckystrike wants to merge 1 commit into
Open
theluckystrike wants to merge 1 commit into
theluckystrike wants to merge 1 commit into
Conversation
Fixes Ericsson#5106. The enqueued_at, started_at, finished_at and last_seen_at columns of BackgroundTask are timezone-less DateTime columns, but config_db_model.py assigned aware datetime.now(timezone.utc) values. psycopg2 sends an aware value as timestamptz and PostgreSQL casts it to timestamp in the session TimeZone, so a database in America/Denver stored Denver wall time. _db_timestamp_to_posix_epoch reads the value back as UTC, so lastHeartbeatEpoch came out 21600 seconds in the past and CodeChecker store raised a false TaskTimeoutError while the task completed on the server. pg8000 and SQLite keep the UTC wall time, and the CI PostgreSQL service runs in UTC, so CI didn't show it. I added _utc_now() in config_db_model.py, which returns the current UTC time without tzinfo, and use it for the six task timestamp assignments. _posix_epoch_to_db_timestamp in api/tasks.py returns naive UTC as well, so the getTasks time filters compare like with like. Column types and migrations stay unchanged. Rows written before this change keep their old values. Those rows belong to tasks that already terminated, nothing reads them to decide a timeout, and every new task row is correct, so there's no migration. On base 29100d4 with only the new test file added, all three new tests fail on the tzinfo check $ cd web && REPO_ROOT=.. BUILD_DIR=../build \ pytest -v server/tests/unit/test_background_task_timestamps.py ...test_abandoned_task_stores_naive_utc FAILED ...test_epoch_db_timestamp_round_trip FAILED ...test_task_transitions_store_naive_utc FAILED E AssertionError: datetime.timezone.utc is not None 3 failed, 1 warning in 1.95s With the fix they pass 3 passed, 1 warning in 0.18s Server unit suite and the CI linters on the fixed tree $ make -C web test_unit_server 64 passed, 1 warning in 2.53s $ make -k pycodestyle pylint exit 0, every pylint run rated 10.00/10 End to end on macOS with PostgreSQL 16.13, timezone America/Denver, CODECHECKER_DB_DRIVER=psycopg2, a server built with BUILD_UI_DIST=NO make package and a one file clangsa analysis $ CodeChecker store reports -j 1 -n e2e \ --url http://localhost:8765/Default base 29100d4 TaskTimeoutError: Task '5a00f8d8...' is still 'RUNNING', but did not have any progress for '6:00:00.411283' (21600.411283 seconds)! store_exit=1 background_tasks status running, last_seen_at 06:58:28, UTC now 12:58:28, the task completed on the server right after with the fix Storing the reports finished successfully. store_exit=0 background_tasks status completed, last_seen_at 12:59:27, UTC now 12:59:33 $ CodeChecker cmd serverside-tasks --url http://localhost:8765 \ --enqueued-after 2026:10:7:12:59:0 --output json one task, enqueuedAt 2026-10-07 12:59:27 $ ... --enqueued-after 2026:10:7:13:0:0 --output json [] The same store against a SQLite product with the fix also finished with exit 0. Not run are make -C web test_matrix_psql_psycopg2 and the functional suites. CI runs them, and its PostgreSQL service runs in UTC, so the end to end check above is the one that covers this bug.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5106
CodeChecker storeon the async path fails with a falseTaskTimeoutErrorwhen the server's PostgreSQL session runs in a non-UTC timezone. The background task columns (enqueued_at,started_at,finished_at,last_seen_at) are timezone-lessDateTime, butBackgroundTaskwrites awaredatetime.now(timezone.utc)values. psycopg2 sends those astimestamptz, and PostgreSQL stores them as local wall time._db_timestamp_to_posix_epochthen reads that local time as UTC, so in America/Denver every heartbeat lands 6 hours in the past and the client gives up on its first poll while the task completes on the server.The fix stores naive UTC, which matches the column type and what SQLite and pg8000 already store.
config_db_model.pyadds_utc_now()(aware UTC now, tzinfo dropped) and uses it for the six task timestamp assignments.api/tasks.pymakes_posix_epoch_to_db_timestampreturn naive UTC too, so thegetTaskstime filters compare like with like.There's no schema change and no migration. Rows written before the upgrade keep their old values, and they belong to tasks that already terminated. Nothing reads them to decide a timeout. CI's PostgreSQL service runs in UTC, which is why the matrix never caught it.
New unit tests in
web/server/tests/unit/test_background_task_timestamps.pycheck that every task transition stores a naive UTC value and that the epoch conversion round trips. On master with only the test file added they fail, and with the fix they pass.End to end I ran a server built with
BUILD_UI_DIST=NO make packageagainst PostgreSQL 16.13 withtimezone = 'America/Denver'andCODECHECKER_DB_DRIVER=psycopg2, then stored a one file clangsa analysis.The same store against a SQLite product on this branch also exits 0. I didn't run
make -C web test_matrix_psql_psycopg2or the functional suites locally. CI runs them, and its PostgreSQL runs in UTC, so the check above is the one that covers this bug.