Skip to content

fix(console): end console sessions when the principal is deleted or its password changes - #704

Open
LeeroyHannigan wants to merge 9 commits into
mainfrom
fix/console-session-revocation
Open

fix(console): end console sessions when the principal is deleted or its password changes#704
LeeroyHannigan wants to merge 9 commits into
mainfrom
fix/console-session-revocation

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What

Console sessions now end when their admin or IAM user is deleted or has its password changed.

Login reads the principal's password hash from the catalog once, verifies the password against that hash, and stores a SHA-256 digest of the same hash in the session. Every console request reads the hash again and compares digests:

  • Same hash: the request goes ahead.
  • Principal deleted, no console password, or a different hash: the session is removed and the request gets the login redirect.
  • Catalog read fails: the request gets HTTP 503 and the session is kept. The refused request does not count as activity, so the 8 hour idle timeout keeps running.

bcrypt draws a new salt for every hash, so every password change stores a new hash, including a change to the same password. The catalog has no record of when a password last changed, and this check does not need one, so there is no migration.

Login used to verify through verify_admin_password and verify_iam_user_password, which read the hash themselves. The console now reads the hash once and uses it for both the check and the digest. A login that races a password change either fails, or creates a session that ends at its first request. It cannot bind the old password to the new hash.

The reads go through a new PasswordHashStore trait, added to CatalogStore. Each backend implements it in its own password_hash_store.rs.

Bound

Nothing is cached. A session stops working at the first request it makes after the delete or password change returns, on any server that shares the catalog. On PostgreSQL that request reads the committed row. MongoDB reads go to the primary. Only one server can open a SQLite catalog.

On Cassandra the hash reads use LOCAL_QUORUM. Deleting an admin, changing an admin password, changing an IAM user password and deleting the user's iam_users row now commit at LOCAL_QUORUM as well; before this change they used the driver default of ONE. Servers in the datacenter that took the write see it at their next request. Servers in other datacenters see it once it has replicated there.

Paths covered

Each of these changes or removes the row the check reads. Apart from the Cassandra write consistency above, none of them needed a code change:

  • DELETE /management/admins/{name} and extenddb manage delete-admin
  • PUT /management/admins/{name}/password and extenddb manage change-admin-password
  • DELETE /management/accounts/{id}/users/{name}, the console's user delete page and extenddb manage delete-user
  • PUT /management/accounts/{id}/users/{name}/password, called by an admin or by the user, and extenddb manage change-user-password

Not covered by this change

Merging with #718

#718 limits failed logins per source address and runs every bcrypt verification through one bounded path. The two PRs conflict in crates/server/src/console/pages/auth_pages.rs and crates/server/src/management/password.rs. #718's body lists what the second of the two to merge has to change, including routing verify_login through the bounded path and removing crate::management::password::verify_password.

Why

A console session stayed valid after its admin or IAM user was deleted or had its password changed. SessionStore::get checked only the 8 hour idle timeout, and nothing except logout removed a session. A cookie issued before the change kept full console access, including access key creation, until it sat idle for 8 hours.

To reproduce on main:

  1. Create an admin with POST /management/admins and {"admin_name": "ops2", "password": "..."}.
  2. Log in to the console as ops2 and keep the extenddb_session cookie.
  3. Delete the admin with DELETE /management/admins/ops2. It returns 204, and a new console login as ops2 fails.
  4. Request /console with the cookie from step 2. The server returns 200 with the dashboard.

Step 3 can also be a password change (PUT /management/admins/ops2/password). Deleting an IAM user or changing its password gives the same result for the user's session.

Testing done

Tests

  • crates/server/src/console/session_tests.rs has 9 tests of SessionStore, 8 of them of SessionStore::validate against an in-memory catalog. They cover a deleted admin next to an admin who keeps access, a removed session that stays removed when the same hash comes back, an admin password change, an IAM user password change and deletion, a catalog error, a 503 that leaves the last-active time unchanged while a passed check refreshes it, a session idle for longer than its time to live, which ends and stays ended while a session idle for less keeps working, and an unknown token. The ninth checks that sessions last 8 hours without activity.
  • crates/server/src/console/pages/mod.rs has 3 tests of the response for each check result: the session data, the login redirect, and HTTP 503.
  • crates/storage-sqlite/src/password_hash_store.rs has 3 tests that the reads follow password changes and deletions, including deletion of the user's account.
  • tests/test_console_session_revocation.py logs in through the console and changes the principal through the management API. A deleted admin's session gets the login redirect while another admin's session still reaches the dashboard. After an admin password change the old session gets the redirect and the new password logs in. An IAM user's session ends on a password change, and the next session ends when the user is deleted. A fourth test logs in on a second server on the same catalog (EXTENDDB_TEST_PEER_ENDPOINT), deletes the admin through the first server, and checks the session on the second server. It is skipped when the variable is unset.

Negative controls

  • With the catalog read removed from SessionStore::validate, 5 of the 6 session tests then in session_tests.rs failed (the unknown token test passed), and the 3 integration tests failed with HTTP 200 where they expect the login redirect. This run came before the activity and idle tests were added.
  • With the last-active refresh moved back into the lookup before the catalog read, the activity test fails.
  • With the 503 mapped to the login redirect, the 503 test fails.
  • With the idle check removed from the lookup, the idle test fails.
  • With SessionStore::new using a time to live other than 8 hours, the 8 hour test fails.

Commands run

Before the rebase onto f49c5f823:

  • cargo fmt --all -- --check and cargo clippy --all-targets -- -D warnings on 1.97.0 for the six changed crates
  • cargo test --locked on stable for extenddb-storage, extenddb-storage-sqlite, extenddb-storage-postgres, extenddb-storage-cassandra, extenddb-storage-mongodb and extenddb-server: 1272 passed, 0 failed
  • cargo test --locked on 1.88.0 for extenddb-storage, extenddb-storage-sqlite and extenddb-server: 545 passed, 0 failed; cargo check --all-targets on 1.88.0 for the other three backends
  • cargo check --locked --workspace --all-targets on 1.97.0
  • tests/test_console_session_revocation.py against a SQLite server: 3 passed, 1 skipped
  • tests/test_console_session_revocation.py against two servers on one PostgreSQL catalog, with EXTENDDB_TEST_PEER_ENDPOINT pointing at the second server: 4 passed, including the test that deletes the admin through one server and checks the session on the other
  • With the PostgreSQL catalog stopped, a console request answered HTTP 503 within 10 ms; after the catalog came back, the same cookie got the dashboard. With the SQLite catalog's write-ahead log index held locked by another process, a console request answered 503 "The catalog is unavailable. Try again." and the session was kept.

CI does not set EXTENDDB_TEST_PEER_ENDPOINT, so it skips the two-server test.

After the rebase onto f49c5f823 and the last two commits (the idle test and an architecture guide sentence):

  • cargo fmt --all -- --check and cargo clippy --locked -p extenddb-server --all-targets -- -D warnings on 1.97.0, the toolchain the Clippy workflow uses: clean
  • cargo test --locked -p extenddb-server on stable (1.99.0): 249 passed, 0 failed
  • cargo test --locked -p extenddb-server on 1.88.0: 249 passed, 0 failed

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

CI ran the workspace tests on Rust 1.88.0 and on stable, and the integration suites on all four backends. All of them passed. Locally I ran the crates listed under Commands run. Clippy ran with -D warnings, without the pedantic group. CatalogStore gains the PasswordHashStore trait, which all four backends implement, and every console request now reads the catalog. Neither has an RFC or an ADR.

ADR / RFC: none

Breaking changes

There are no configuration or schema changes. These behaviours change:

  • Changing your own password through the management API or the CLI ends your console session. Log in again with the new password.
  • Each console request makes one primary key read of admin_users or iam_users. With 50 clients loading the dashboard in a loop, debug builds on a busy host served 22 to 28 percent fewer pages per second than main on SQLite and 31 to 37 percent fewer on PostgreSQL. Item operations use a separate connection pool, and their latency under that load moved the same way on main and on this branch.
  • When that read fails, the console returns HTTP 503. Before, the session check did not read the catalog.
  • On Cassandra, the four writes listed under Bound need a quorum of replicas in the local datacenter. With fewer replicas up they now fail, where before they succeeded at ONE.

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

@LeeroyHannigan
fix(storage): add password hash reads for console session checks
bd2048f 
Add a PasswordHashStore trait to CatalogStore with two reads: the
stored password hash of an admin, and the console password hash of an
IAM user. Each backend implements it in a new password_hash_store.rs.
The console uses these reads to tie a session to the password hash it
was created against.

On Cassandra the reads run at LOCAL_QUORUM. Deleting an admin and
changing an admin's password now write at LOCAL_QUORUM instead of the
driver default of ONE. Changing an IAM user's password and deleting
its iam_users row now commit their lightweight transactions at
LOCAL_QUORUM through a new apply_lwt_quorum helper. A read issued
after one of these writes returns sees it in the same datacenter.
@LeeroyHannigan
fix(console): end sessions of deleted principals and changed passwords
9378114 
A console session stayed valid after its admin or IAM user was
deleted or had its password changed, until it had been idle for 8
hours. SessionStore::get checked only the idle timeout, and nothing
but logout removed a session, so the old cookie kept full console
access.

Login now reads the principal's password hash once, verifies the
password against it, and binds the session to a SHA-256 digest of
that hash. Every console request reads the hash from the catalog
again. When the principal is gone, has no console password, or has a
different hash, the session is removed and the request is redirected
to the login page. bcrypt draws a new salt for every hash, so every
password change stores a different hash. When the catalog cannot be
read, the request gets a 503 and the session is kept.

All servers read the same catalog, so a deletion or password change
made through one server ends the session on every server, at the next
request that server serves for it.
@LeeroyHannigan
test(console): cover session end on principal delete and password change
a1dfb20 
Log in through the console, then change the principal through the
management API and check the next console request. A deleted admin's
session gets the login redirect while another admin's session still
reaches the dashboard. After an admin password change the old session
gets the login redirect and the new password logs in. An IAM user's
session ends on a password change and again on deletion.

A fourth test logs in on a second server that shares the catalog
(EXTENDDB_TEST_PEER_ENDPOINT), deletes the admin through the first
server, and checks the session on the second one. It is skipped when
the variable is unset.
@LeeroyHannigan
docs: describe how console sessions end on principal changes
bf0c553 
The security model, architecture guide, admin guide and getting
started guide now say that deleting an admin or IAM user, or changing
its password, ends its console sessions at their next request on every
server. The security model states the Cassandra datacenter bound and
the 503 response when the catalog cannot be read, and the
troubleshooting guide lists both causes.
@LeeroyHannigan
fix(console): count only a passed session check as activity
55f90e9 
The session lookup refreshed a session's last-active time before the
catalog read. A request refused with 503 because the catalog could not
be read therefore still counted as activity, and a client retrying
through a long catalog outage kept an idle session alive for as long
as the outage lasted.

The lookup now only checks the idle timeout. A request whose check
passes refreshes the time after the catalog read. A request refused
with 503 leaves it unchanged, so the 8 hour idle timeout keeps running
during an outage.
@LeeroyHannigan
test(console): check the response for each session check result
53b3017 
require_session turns the result of the session check into the
session data, the login redirect, or a 503 when the catalog cannot be
read. That mapping is now its own function, with one test per result.
No test covered the 503 before: mapping it to the login redirect
passed every test.
@LeeroyHannigan
docs: say how console sessions idle out and when a reload still answe…
78e2c7e 
…rs 503

The architecture guide now says sessions expire after 8 hours of
inactivity, as the other guides do, and names the PasswordHashStore
trait the console reads password hashes through. The security model
says a request refused with 503 does not count as activity. The
troubleshooting entry for the 503 says that after a PostgreSQL restart
the first reloads can still answer 503, once for each pooled catalog
connection the restart closed.
@LeeroyHannigan
test(console): end a session that stays idle longer than its time to …
7a9a692 
…live

No test covered the idle timeout in SessionStore::lookup: with the check
removed, an idle session never ended and every console test still passed.

SessionStore now holds its time to live in a field. new() sets it to
SESSION_TTL, 8 hours, and a test-only with_ttl() sets a shorter one, so a
test can backdate a session past the limit. Subtracting 8 hours from
Instant::now() returns None on Windows when the host has been up for less
than 8 hours.

One new test checks that a session idle for longer than its time to live
ends and stays ended on the next request, while a session idle for less
keeps working. The other checks that new() keeps the 8 hour limit.
@LeeroyHannigan
docs: name the four store traits the console reads through
9d89da8 
The architecture guide said the console reads all data through
ManagementStore and PasswordHashStore. It also reads settings through
SettingsStore and the login limits through RateLimitStore. The sentence
now names all four, which are part of CatalogStore.
@LeeroyHannigan LeeroyHannigan added bug Something isn't working security labels Oct 31, 2026
This was referenced Nov 1, 2026
Open
Open
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amrith amrith Awaiting requested review from amrith amrith is a code owner
@c33howard c33howard Awaiting requested review from c33howard c33howard is a code owner
@jcshepherd jcshepherd Awaiting requested review from jcshepherd jcshepherd is a code owner
@pdf-amzn pdf-amzn Awaiting requested review from pdf-amzn pdf-amzn is a code owner
@yesyayen yesyayen Awaiting requested review from yesyayen yesyayen is a code owner
@robinnsc robinnsc Awaiting requested review from robinnsc robinnsc is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

bug Something isn't working security

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@LeeroyHannigan