Conversation
… facts Replaces the unconditional newest-wins conflict rule in the sleeptime reflection prompt with a structured Durability and Resolutions policy: - Update only when newer evidence explicitly establishes staleness - Annotate conflicting-but-unverified evidence as a separate fact - Restraint: never weaken safety-critical facts on anecdote Measured on a 120-run eval matrix (4 conflict cases x 3 Bedrock models x repeats): correct conflict handling improved from 90/120 to 110/120, with the largest gains on safety-critical overwrite cases. Addresses letta-ai#4029
Tusharv23
force-pushed
the
feature/sleeptime-conflict-resolution
branch
from
August 28, 2026 10:39
06f0d66 to
fd9e982
Compare
Author
|
Hey @kl2806, checking in on this PR. Since submitting I've re-run the sanity set against the final wording (11/12, results in the PR body) and added confidence intervals to the underlying eval. Happy to make any changes if the wording or scope needs adjusting, is there anything blocking a review? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #4029.
The sleeptime reflection prompt currently instructs the subagent to "resolve contradictions in favor of the latest evidence" unconditionally. This newest-wins rule causes destructive memory overwrites: a single unverified or anecdotal observation can replace a confirmed durable fact, including safety-critical ones (e.g., a confirmed severe allergy softened to "mild" after one anecdote about eating shellfish without reaction).
This PR replaces the inline conflict clause with a structured Durability and Resolutions policy: update only when newer evidence explicitly establishes staleness; otherwise record the conflicting observation as a separate fact and leave the original unchanged; never weaken confirmed safety-critical facts on anecdote.
Prompt-only change (
src/agent/subagents/builtin/reflection.md); no code touched.Tradeoff: memory grows slightly faster (conflicts are annotated instead of overwritten), in exchange for not destroying confirmed facts on weak evidence.
Verification
bun teston this branch: 6872 pass / 46 fail, identical counts to cleanmainon the same machine (pre-existing environment-dependent failures: telemetry auth, PTY, stream-json integration). No new failures introduced.AI Disclosure
AI Tool(s) Used
GitHub Copilot (Claude Fable 5) - used to help draft the prompt wording, plan the eval harness. All runs, changes and eval results done by me.
Human Verification
I have reviewed and understand every change in this pull request and take responsibility for its correctness.