@@ -16,23 +16,89 @@ answer's text alone, keep the `ContentText` blocks of the turn's `AssistantMessa
1616[ judge step's] ( language-models.mdx#a-judge-step ) ` answer_text ` does, and keep the
1717structured rollout for later inspection.
1818
19- A model turn made by an API-backed session records the provider's response in
20- ` ModelTurn.metadata["response"] ` , as JSON, including its stop reason and usage. Metadata is defined
21- by whatever produced the turn, so validate it before scoring with it.
19+ ## Metadata
20+
21+ Turns (` ContextTurn ` , ` ModelTurn ` , ` UserTurn ` , ` ToolTurn ` ), ` EndRollout ` , and ` ChatRollout ` carry
22+ ` metadata ` : a tuple of ` DynamicData ` entries, each holding one
23+ [ tagged dataclass] ( ../../../fxtr/references/concepts/entities-and-custom-types.mdx#inline-tagged-unions ) in its stored
24+ form. Whatever produced the turn decides which types it records, so a reader names the type it
25+ expects. ` find_metadata(metadata, SomeType) ` returns the first entry of that type, decoded, or
26+ ` None ` if there is none; ` entry.try_as_type(SomeType) ` does the same for a single entry. An entry
27+ tagged with the type but holding fields that don't match it raises ` ConversionError ` .
28+
29+ A model turn made by an API-backed session records the provider's response as a ` ModelResponse ` ,
30+ including its stop reason and usage:
31+
32+ ``` python
33+ from behaviors.types import ModelResponse, find_metadata
34+
35+ response = find_metadata(model_turn.metadata, ModelResponse)
36+ if response is not None :
37+ print (response.stop_reason.type, response.usage.output_tokens)
38+ ```
39+
40+ To record something of your own, such as which question a context turn asked, define a frozen
41+ tagged dataclass with a type ID in a domain you control, and wrap each value with
42+ ` DynamicData.from_tagged ` :
43+
44+ ``` python
45+ from dataclasses import dataclass
46+
47+ from behaviors.types import ContentText, ContextTurn, DynamicData, UserMessage, find_metadata
48+ from fxtr.entity_defns.dataclass_bases import TaggedDataclass
49+
50+
51+ @dataclass (frozen = True )
52+ class QuestionIndex (TaggedDataclass , fxtr_type = " org.example.my_experiment.QuestionIndex.v1"
53+ index: int
54+
55+
56+ turn = ContextTurn(
57+ messages = (UserMessage(content = (ContentText(text = " What is 2 + 2?"
58+ available_tools = None ,
59+ metadata = (DynamicData.from_tagged(QuestionIndex(index = 0 )),),
60+ )
61+ assert find_metadata(turn.metadata, QuestionIndex) == QuestionIndex(index = 0 )
62+ ```
63+
64+ Add an entry to existing metadata with ` turn.metadata + (DynamicData.from_tagged(value),) ` .
65+ ` Transcript ` , ` TranscriptGroup ` , and ` AgentRun ` metadata is different: it is a JSON object with
66+ string keys, matching [ Docent] ( docent-readings.mdx ) 's metadata.
2267
2368## Storing records
2469
2570` ChatRollout ` , ` Transcript ` , ` TranscriptGroup ` , ` AgentRun ` , and the concrete session
2671initializations, such as ` SetSystemMessage ` , are fxtr entities, with type IDs of the form
27- ` org.transluce.behaviors.<Class>.v1 ` . A step that returns a ` ChatRollout ` has it stored like any
28- other entity (see [ Entities and custom types] ( ../../../fxtr/references/concepts/entities-and-custom-types.mdx ) ), and
72+ ` org.transluce.behaviors.<Class>.v<N> ` , such as ` org.transluce.behaviors.ChatRollout.v2 ` . A
73+ version is bumped when a type's stored shape changes, and a record stored under an earlier version
74+ doesn't load as the current type. Versions are per type: ` ChatRollout ` and its turns use ` .v2 ` ,
75+ while messages and ` ModelResponse ` use ` .v1 ` . Only the current version of each type is supported.
76+
77+ A step that returns a ` ChatRollout ` has it stored like any other entity
78+ (see [ Entities and custom types] ( ../../../fxtr/references/concepts/entities-and-custom-types.mdx ) ), and
2979` await storage.store(entity) ` and ` await storage.load(ref) ` work on them too. To load a bare ID,
3080pass the concrete type, as in ` as_type=ChatRollout ` , not an abstract base such as
3181` SessionInitialization ` . A tree of transcripts and groups is stored as one entity, so there is no
3282need to store each child separately.
3383
84+ A stored rollout includes every field, including ` initialization ` . It keeps its initialization
85+ inline, as ` DynamicData ` (or an explicit null if there was none), since initialization types are
86+ open-ended. In Python, read it by naming the type you handle, which gives ` None ` for
87+ any other type:
88+
89+ ``` python
90+ from behaviors.types import SetSystemMessage
91+
92+ init = rollout.initialization
93+ set_system = init.try_as_type(SetSystemMessage) if init is not None else None
94+ ```
95+
3496fxtr checks stored records strictly: give fields their declared types (` temperature=0.0 ` , not
35- ` 0 ` ), and keep metadata to JSON values with string keys.
97+ ` 0 ` ), and keep transcript metadata to JSON values with string keys.
98+
99+ The step cache can still hold results written under another type version. When changing the
100+ rollout version a project uses, launch under a fresh root or clear the affected steps' cache
101+ entries before rerunning (see [ Managing the step cache] ( ../../../fxtr/references/concepts/managing-the-step-cache.mdx ) ).
36102
37103## Transcripts
38104
@@ -69,7 +135,11 @@ A new project's renderer package doesn't depend on it yet: add
69135## Checkpointing sessions
70136
71137A serializable session can save its state and be restored from it. ` await session.get_state() `
72- returns a JSON object, and ` await factory.restore_session(state) ` starts a session in that state.
138+ returns the state as ` DynamicData ` , tagged with the implementation's own state type, and
139+ ` await factory.restore_session(state) ` starts a session in that state; it raises ` ConversionError `
140+ for a state that isn't its own. To keep a state as JSON, write
141+ ` json.dumps(thaw_fxtr_data(state.data)) ` (` thaw_fxtr_data ` is in ` fxtr.core.entity_data ` ), and
142+ read it back with ` DynamicData.from_data(json.loads(text)) ` .
73143The sessions of ` chat_session_model_from_api ` and of ` ScriptedUserSimulator ` support this, and the
74144` serializable_context_policy_from_* ` combinators build checkpointable policies when every component
75145they combine is checkpointable. Generator-based policies can't be checkpointed, because a
0 commit comments