diff --git a/.donna/project/work/polish.md b/.donna/project/work/polish.md index aeb1c0f9..3214fe85 100644 --- a/.donna/project/work/polish.md +++ b/.donna/project/work/polish.md @@ -5,7 +5,7 @@ kind = "donna.lib.workflow" start_operation_id = "run_autoflake_script" ``` -Initiate operations to polish and refine the donna codebase: running & fixing tests, formatting code, fixing type annotations, etc. +Initiate operations to polish and refine the donna codebase: running & fixing tests, formatting code, fixing type annotations, etc. This workflow MUST NOT be used to introduce new logic into the project or refactor it — only to fix existing issues. ## Run Autoflake diff --git a/README.md b/README.md index 6e045dc7..55e5f970 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ Your agent will generate [state machines](https://en.wikipedia.org/wiki/Finite-s Donna allows your agent to execute hundreds of consecutive steps without swaying away from the defined workflow. Branching, loops, nested calls, and recursion — all possible. +![Journal log demonstration](./docs/images/journal-demo.gif) + ## What is Donna? Donna is a CLI tool that helps coding agents like Codex focus on the task at hand by keeping high-level control flow in explicit Donna workflows. Donna dictates what should be done at each step of the work, so the agent can focus on the actual piece. @@ -155,11 +157,18 @@ Donna will: **Donna is a CLI tool for agents.** You rarely need to use it directly. +However, it is convenient to run `donna journal view --follow` in a separate terminal to see what is going on in the current session. + Commands you may need: -- `donna -p human workspaces init` — Initialize Donna in your project. -- `donna -p human sessions start` — start a new working session, remove everything from the previous session. -- `donna -p human artifacts list ` — list artifacts with short descriptions. +- `donna workspaces init` — Initialize Donna in your project. +- `donna sessions start` — start a new working session, remove everything from the previous session. +- `donna artifacts list ` — list artifacts with short descriptions. +- `donna journal view [--lines N] [--follow]` — view the log of work performed in the current session. + +Here is an example of the real Donna session work log: + +![Journal log demonstration](./docs/images/journal-demo.gif) Use `donna --help` for a quick reference. @@ -194,7 +203,7 @@ Note that the default Donna workflows are designed to be reliable and useful for Points of interest: - [donna:rfc:specs:request_for_change](./donna/artifacts/rfc/specs/request_for_change.md) — specification of the RFC document. -- [donna:rfc:work:create](./donna/artifacts/rfc/work/create.md) — workflow to create a RFC document. +- [donna:rfc:work:request](./donna/artifacts/rfc/work/request.md) — workflow to create a RFC document. - [donna:rfc:work:plan](./donna/artifacts/rfc/work/plan.md) — workflow to plan work on an RFC — creates a new workflow. - [donna:rfc:work:do](./donna/artifacts/rfc/work/do.md) — meta workflow to automate the whole work from a developer request to a changelog update. diff --git a/changes/next_release.md b/changes/next_release.md new file mode 100644 index 00000000..0421ab0b --- /dev/null +++ b/changes/next_release.md @@ -0,0 +1,13 @@ + +### Changes + +- gh-35 — Implemented `donna journal` CLI subcommands for writing and viewing session journal. + - Added journaling of all significant actions and events. + - Removed `single_mode` formatting option from protocol formatter APIs. + - Command `donna artifacts validate` now accepts an artifact pattern instead of a single artifact id. + - Removed `donna artifacts validate-all` command. +- gh-41 — Added `Design` step into RFC workflow. + - Added `donna:rfc:specs:design` and `donna:rfs:work:design`. + - Updated `donna:rfc:work:do` to include `design` step between `create` and `plan`. + - Renamed `donna:rfc:work:create` to `donna:rfc:work:request`. +- gh-58 — Unified error handling in `donna.machine.sessions` and `donna.cli.*` modules. diff --git a/changes/unreleased.md b/changes/unreleased.md deleted file mode 100644 index 373eb997..00000000 --- a/changes/unreleased.md +++ /dev/null @@ -1,2 +0,0 @@ - -No changes. diff --git a/docs/images/journal-demo.gif b/docs/images/journal-demo.gif new file mode 100644 index 00000000..a31e5745 Binary files /dev/null and b/docs/images/journal-demo.gif differ diff --git a/donna/artifacts/intro.md b/donna/artifacts/intro.md index 8caf4d86..5b871fd8 100644 --- a/donna/artifacts/intro.md +++ b/donna/artifacts/intro.md @@ -36,3 +36,24 @@ Artifact type tags: 4. If you are executing a workflow operation and need to perform a complex action or changes, you SHOULD search for an appropriate workflow and run it as a child workflow — it is the intended way to use Donna. 5. Run to list all workflows: `{{ donna.lib.list("**", tags=["workflow"]) }}` 6. Run to list all specifications: `{{ donna.lib.list("**", tags=["specification"]) }}` + +## Journaling + +You MUST use `donna journal write` to track your actions and thoughts, according the description in `{{ donna.lib.view("donna:usage:cli") }}`. + +Journaling is a required part of workflow execution. An action request MUST be considered incomplete until required journal records are written. + +Journaling lifecycle for each non-trivial action request: + +1. Start intent (`Goal:`) before substantial work begins. +2. Progress updates (`Step:`) at significant phase boundaries. +3. Concrete edits (`Change:`) after meaningful source/artifact update batches. +4. Completion handoff (`Step:`) before calling `sessions action-request-completed`. + +Journal records MUST be change/decision-oriented and SHOULD be sufficient for another agent to continue work without re-discovery. + +If you perform a long operation (e.g., exploring the codebase, designing a solution) that takes more than 10 seconds, you MUST journal your progress. + +You MUST use `donna journal view --lines 100` to read the last records after you compress your context. + +If your work is interrupted and you resume later, you MUST first journal `Resume context and next action`. diff --git a/donna/artifacts/rfc/specs/design.md b/donna/artifacts/rfc/specs/design.md new file mode 100644 index 00000000..020d0422 --- /dev/null +++ b/donna/artifacts/rfc/specs/design.md @@ -0,0 +1,141 @@ +# Format of the Design document + +```toml donna +kind = "donna.lib.specification" +``` + +This document describes the format and structure of a Design document used to design changes to a project proposed in a Request for Change (RFC). This document is an input for planning and will be treated as strong recommendations for the implementation of the proposed change. + +## Overview + +Donna introduces a group of workflows located in `donna:rfc:*` namespace that organize the process of proposing, reviewing, and implementing changes to a project via RFC and Design documents. + +You create a Design document to explicitly describe the exact changes you want to make to the project in order to implement the RFC. + +If not otherwise specified, Design documents for the session MUST be stored as `session:design:` artifacts in the session world. + +**The Design document MUST list exact changes to the project that will be implemented.** E.g. concrete function names and signatures, file paths, data structures, etc. + +**The Design document MAY skip implementation details.** E.g. it may skip the exact implementation of a function body, but it should specify the function signature and its purpose; It may use pseudocode to describe the logic of a function, but it should not skip describing the logic at all. + +The Design document MUST NOT be a high-level description of the problem and solution. A high-level description of the problem and solution should be provided in the RFC document. + +**Identifiers in this document MUST follow project-specific naming conventions.** Before working on a Design document, you should read the artifacts with project guidelines for naming and code style, if any exist. + +## Design document structure + +The RFC document is Donna artifact (check `{{ donna.lib.view("donna:usage:artifacts") }}`) with the next structure: + +- **Primary section** — title and short description of the proposed change. +- **Inputs** — list of input documents that are relevant for the proposed change, starting from the RFC document. +- **Architecture changes** — list of high-level architectural changes that MUST be implemented. +- **Highlevel code changes** — list of high-level code changes that MUST be implemented: modules to add/modify, functions to add/modify, classes to add/modify, etc. This section should not include low-level implementation details, but it should include enough details to understand the scope of the change and its impact on the project. +- **Data structure changes** — list of changes to data structures that MUST be implemented. +- **Interface changes** — list of changes to interfaces (e.g. function signatures, API endpoints, etc.) that MUST be implemented. +- **Tests changes** — list of autotests that MUST be implemented/updated/removed. Only if the project already uses autotests. +- **Documentation changes** — list of changes in the documentation. Only if the project already has documentation. +- **Other changes** — list of other changes that do not fit into the previous sections, but are still relevant for the proposed change. +- **Order of implementation** — a proposed order of implementation of the changes listed in the previous sections. This section should be treated as a recommendation (from the author of the Design document), not a strict requirement. + +## General language and format + +- You MUST follow [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119.txt) for keywords like MUST, SHOULD, MAY, etc. +- You MUST follow `{{ donna.lib.view("donna:usage:artifacts") }}`. +- You MUST follow the structure specified in this document. + +### List format + +- If a section is described as a list, it MUST contain only a single markdown list. +- Each list item MUST be concise and clear. +- Each list item SHOULD be atomic and focused on a single aspect. +- Reviewer MUST be able to tell whether the list item statement is true or false by inspecting the resulting artifacts and behavior. + +Examples: + +- Bad: `- Implement a new module.` +- Bad: `- The interface MUST include button A, button B, and dropdown C.` +- Good: `- Implement a module "a.b.c" that is responsible for X, Y, and Z.` +- Good: `- Add a button class "ButtonA" to the module "a.b.c"` +- Good: `- "ButtonA" class MUST have a method "click()" that performs action X when called.` + +Common approaches to improve list items: + +- Split a single item with an enumeration into multiple items with a single focus. +- Transform an abstract item into a concrete one by specifying measurable criteria or user-visible behavior. + +## `Primary` section + +- Title MUST be concise and reflect the essence of the proposed change. Derive it from the RFC. +- Description MUST provide an essence of the proposed change. + +## `Inputs` section + +- The section MUST contain a list of documents that are relevant to the proposed change. +- The first item in the list MUST be the original RFC document that describes the problem and the proposed solution. +- Other items in the list MAY include other documentation, code files, external links, etc., that are relevant for the proposed change. + +## `Architecture changes` section + +- The section MUST contain a free-form but precise and grounded description of the high-level architectural changes that MUST be implemented. +- Along the text, you may add code snippets, diagrams, and other visual aids to make the description clearer and more precise. + +## `High-level code changes` section + +- The section MUST contain a list of high-level changes in the code. +- The level of abstraction: `add a module A`, `remove the class B`, `change the implementation of a function C`. +- The section MUST list only the most important changes that are significant cornerstones of the proposed change. +- The section MAY omit low-level details, such as the small or utilitarian functions, minor refactorings, etc. + +## `Data structure changes` section + +- The section MUST list exact changes to data structures that MUST be implemented. +- Each change MUST be accompanied by a description of the purpose of the change and its impact on the project. + +Examples of statements about structure changes: + +- Bad: `- Change the data structure of the project.` +- Bad: `- Update the class A.` +- Good: `- Add a field "x" to the class "A".` +- Good: `- Change the type of the field "y" in the class "B" from "int" to "str".` +- Good: `- Add structure "C" with fields "a", "b", and "c".` + +## `Interface changes` section + +- The section MUST list exact changes to interfaces that MUST be implemented. +- Each change MUST be accompanied by a description of the purpose of the change and its impact on the project. + +Examples of statements about interface changes: + +- Bad: `- Change the interface of functions in the project.` +- Bad: `- Update the API endpoints.` +- Good: `- Add a new API endpoint "/api/v1/resource" that accepts POST requests with JSON body containing fields "a", "b", and "c".` +- Good: `- Change the signature of the function "foo" from "foo(x: int) -> str" to "foo(x: int, y: str) -> str".` + +## `Tests changes` section + +- If the project does not use autotests, this section MUST contain a statement `No changes in tests are required, since the project does not use autotests.` +- If the project uses autotests, this section MUST contain a list of autotests that MUST be implemented/updated/removed. +- Each changes piece of logic MUST have at least one corresponding autotest that verifies its correctness and prevents regressions in the future. +- Each added/updated branch of logic MUST have at least one corresponding autotest that verifies its correctness and prevents regressions in the future. + +Examples of statements about test changes: + +- Bad: `- Add tests for the new module.` +- Bad: `- Update tests for the changed function.` +- Good: `- Add a test "test_foo_with_valid_input" that verifies that the function "foo" returns the expected result when called with valid input.` +- Good: `- Add a test "test_foo_success_path" that verifies that the function "foo" returns the expected result when called with input that follows the success path.` + +## `Documentation changes` section + +- If the project does not have documentation, this section MUST contain a statement `No changes in documentation are required, since the project does not have documentation.` +- If the project has documentation, this section MUST contain a list of changes in the documentation that MUST be implemented. + +## `Other changes` section + +- The section MAY contain a list of other changes that do not fit into the previous sections, but are still relevant for the proposed change. +- The section MUST be a single statement: `No other changes are required, since all relevant changes are covered in the previous sections.` if there are no other changes to mention. + +## `Order of implementation` section + +- The section MUST contain a proposed order of implementation of the changes listed in the previous sections. +- The section MUST refer only to identifiers mentioned in the previous sections, and it MUST NOT introduce new identifiers or entities that are not mentioned in the previous sections. diff --git a/donna/artifacts/rfc/specs/request_for_change.md b/donna/artifacts/rfc/specs/request_for_change.md index a0f13c54..9c56893a 100644 --- a/donna/artifacts/rfc/specs/request_for_change.md +++ b/donna/artifacts/rfc/specs/request_for_change.md @@ -4,20 +4,16 @@ kind = "donna.lib.specification" ``` -This document describes the format and structure of a Request for Change (RFC) document used to propose changes to a project by Donna workflows from `donna:rfc:*` namespace. +This document describes the format and structure of a Request for Change (RFC) document used to propose changes to a project by Donna workflows from `donna:rfc:*` namespace. This document is an input for a Design document creation. ## Overview -Donna introduces a group of workflows located in `donna:rfc:*` namespace that organize the process of proposing, reviewing, and implementing changes to a project via RFC documents. +Donna introduces a group of workflows located in `donna:rfc:*` namespace that organize the process of proposing, reviewing, and implementing changes to a project via RFC and Design documents. You create RFC documents to propose changes to the project. If not otherwise specified, RFC documents for the session MUST be stored as `session:rfc:` artifacts in the session world. -The agent (via workflows) creates the artifact and polishes it iteratively as the RFC process progresses. - -After the RFC is completed, the agent (via workflows) MAY implement changes directly or create and execute a workflow based on the RFC content. - ## RFC structure The RFC document is Donna artifact (check `{{ donna.lib.view("donna:usage:artifacts") }}`) with the next structure: diff --git a/donna/artifacts/rfc/work/design.md b/donna/artifacts/rfc/work/design.md new file mode 100644 index 00000000..54ec9bb5 --- /dev/null +++ b/donna/artifacts/rfc/work/design.md @@ -0,0 +1,121 @@ +# Create a Design document + +```toml donna +kind = "donna.lib.workflow" +start_operation_id = "start" +``` + +This workflow creates a Design document artifact based on an RFC and aligned with `donna:rfc:specs:design`. + +## Start Work + +```toml donna +id = "start" +kind = "donna.lib.request_action" +fsm_mode = "start" +``` + +1. Read the specification `{{ donna.lib.view("donna:rfc:specs:design") }}` if you haven't done it yet. +2. Read the specification `{{ donna.lib.view("donna:usage:artifacts") }}` if you haven't done it yet. +3. `{{ donna.lib.goto("ensure_rfc_artifact_exists") }}` + +## Ensure RFC artifact exists + +```toml donna +id = "ensure_rfc_artifact_exists" +kind = "donna.lib.request_action" +``` + +At this point, you SHOULD have a clear RFC to design. + +1. If you have an RFC artifact id in your context, view it and `{{ donna.lib.goto("prepare_design_artifact") }}`. +2. If you have no RFC artifact id in your context, but you know it is in one of `{{ donna.lib.list("session:**") }}` artifacts, find and view it. Then `{{ donna.lib.goto("prepare_design_artifact") }}`. +3. If you have no RFC artifact id in your context, and you don't know where it is, ask the developer to provide the RFC artifact id or to create a new RFC. After you get it and view the artifact, `{{ donna.lib.goto("prepare_design_artifact") }}`. + +## Prepare Design artifact + +```toml donna +id = "prepare_design_artifact" +kind = "donna.lib.request_action" +``` + +1. If the name of the artifact is not specified explicitly, assume it to be `session:design:`, where `` SHOULD correspond to the RFC slug. +2. Save the next template into the artifact, replace `` with appropriate values. + +~~~ +# + +```toml donna +kind = "donna.lib.specification" +``` + +<short description of the proposed design> + +## Inputs + +- <RFC artifact id> + +## Architecture changes + +## High-level code changes + +## Data structure changes + +## Interface changes + +## Tests changes + +## Documentation changes + +## Other changes + +## Order of implementation +~~~ + +3. `{{ donna.lib.goto("initial_fill") }}` + +## Initial Fill + +```toml donna +id = "initial_fill" +kind = "donna.lib.request_action" +``` + +1. Read the specification `{{ donna.lib.view("donna:rfc:specs:design") }}` if you haven't done it yet. +2. Read the RFC artifact selected in the previous step if you haven't done it yet. +3. Analyze the project if needed to understand the requested change context. +4. Fill in all sections of the Design draft artifact. +5. Ensure the first item in `Inputs` section is the RFC artifact id. +6. `{{ donna.lib.goto("review_design_format") }}` + +## Review Design Format + +```toml donna +id = "review_design_format" +kind = "donna.lib.request_action" +``` + +1. List mismatches between the Design artifact and the Design specification `{{ donna.lib.view("donna:rfc:specs:design") }}`. +2. For each mismatch, make necessary edits to the Design draft artifact to ensure compliance. +3. `{{ donna.lib.goto("review_design_content") }}` + +## Review Design Content + +```toml donna +id = "review_design_content" +kind = "donna.lib.request_action" +``` + +1. Read the Design document and identify gaps, inconsistencies, or areas for improvement in accordance with the RFC and current project context. Use `{{ donna.lib.view("donna:research:work:research") }}` workflow if you need to make a complex decision. +2. Make necessary edits to the Design draft artifact to address identified issues. +3. If there were changes made on this step or the previous `review_design_format` step `{{ donna.lib.goto("review_design_format") }}`. +4. If no changes were made, `{{ donna.lib.goto("finish") }}`. + +## Complete Draft + +```toml donna +id = "finish" +kind = "donna.lib.finish" +``` + +Design draft is complete and ready for implementation planning. diff --git a/donna/artifacts/rfc/work/do.md b/donna/artifacts/rfc/work/do.md index cfb2e125..c39d9294 100644 --- a/donna/artifacts/rfc/work/do.md +++ b/donna/artifacts/rfc/work/do.md @@ -9,8 +9,9 @@ start_operation_id = "estimate_complexity" This workflow uses a description of a problem or changes required from the developer or parent workflow to: 1. Create a Request for Change (RFC) artifact. -2. Plan the work required to implement the RFC. -3. Execute the planned work. +2. Create a Design document artifact based on the RFC. +3. Plan the work required to implement the designed changes. +4. Execute the planned work. ## Estimate complexity @@ -53,7 +54,18 @@ fsm_mode = "start" 1. Choose a workflow to create an RFC artifact. 2. Run the chosen workflow. -2. After completing the workflow `{{ donna.lib.goto("plan_rfc_work") }}`. +3. After completing the workflow `{{ donna.lib.goto("design") }}`. + +## Design + +```toml donna +id = "design" +kind = "donna.lib.request_action" +``` + +1. Choose a workflow to create a Design document artifact based on the RFC created in the previous step. +2. Run the chosen workflow. +3. After completing the workflow `{{ donna.lib.goto("plan_rfc_work") }}`. ## Plan RFC work @@ -62,7 +74,7 @@ id = "plan_rfc_work" kind = "donna.lib.request_action" ``` -1. Choose the workflow to plan the work required to implement the RFC created in the previous step. +1. Choose the workflow to plan the work. If you created a Design document in the previous step, use it as a basis. 2. Run the chosen workflow. 3. Ensure you know the workflow id created in the previous step (default is `session:execute_rfc` if not specified). 4. After completing the workflow `{{ donna.lib.goto("execute_rfc_work") }}`. diff --git a/donna/artifacts/rfc/work/plan.md b/donna/artifacts/rfc/work/plan.md index 5f754b3f..180ab8dd 100644 --- a/donna/artifacts/rfc/work/plan.md +++ b/donna/artifacts/rfc/work/plan.md @@ -1,12 +1,12 @@ -# Plan work on Request For Change +# Plan work on base of Design & Request for Change documents ```toml donna kind = "donna.lib.workflow" start_operation_id = "start" ``` -This workflow plans the work required to implement a specified Request for Change (RFC). The result of this workflow is a new workflow in the `session:*` world with detailed steps to implement the RFC. +This workflow plans the work required to implement a specified Design document. The RFC document SHOULD be used as a helper context. The result of this workflow is a new workflow in the `session:*` world with detailed steps to implement the designed changes. ## Start Work @@ -16,9 +16,10 @@ kind = "donna.lib.request_action" fsm_mode = "start" ``` -1. Read the RFC that the developer or parent workflow wants you to implement. -2. Read the specification `{{ donna.lib.view("donna:usage:artifacts") }}` if you haven't done it yet. -2. `{{ donna.lib.goto("prepare_workflow_artifact") }}` +1. Read the Design document that the developer or parent workflow wants you to implement. +2. Read the RFC document that the developer or parent workflow wants you to implement, if it exists. +3. Read the specification `{{ donna.lib.view("donna:usage:artifacts") }}` if you haven't done it yet. +4. `{{ donna.lib.goto("prepare_workflow_artifact") }}` ## Prepare workflow artifact @@ -27,10 +28,11 @@ id = "prepare_workflow_artifact" kind = "donna.lib.request_action" ``` -1. If the name of the artifact is not specified explicitly, assume it to `session:plans:<short-id-equal-to-rfc-slug>`. +1. If the name of the artifact is not specified explicitly, assume it to `session:plans:<short-id-equal-to-design-slug>`. 2. Create a workflow with the next operations: - Start - - A step for each action point in the RFC, ordered to minimize dependencies between steps and introduce changes incrementally. + - A step for each action point in the RFC document and each item in the `Order of implementation` in Design document with the goal to minimize dependencies between steps and introduce changes incrementally. + - Add additional steps if the design requires it. - (`id="review_changes"`) A gate step to start reviewing the changes. - A gate step to start reviewing the deliverables. - A step per deliverable to check if it exists and is correct. If the deliverable is missing or incorrect, the step MUST specify how to fix it and `goto` to the `review_changes` step. @@ -52,6 +54,12 @@ For each of the steps in the workflow you created: 2. If the step requires both research and implementation, split it into two steps: research step and implementation step. 3. If the step required editing multiple artifacts (multiple files, multiple functions, etc.), split it into multiple steps, one per change required. 4. If the step is too big or complex, split it into multiple smaller steps. +5. Fux & improve naming and IDs in the step's title and body. + +Naming rules: + +- A title of operation MUST always have meaning without knowing context: no sequential numbers, no generic titles like `Plan part 1`. +- If you mention an ID from the RFC or the Design document, you MUST include a human-readable name of the entity you refer. E.g. not `O1` but `O1 (Implement a new module "a.b.c")` After all steps are reviewed: @@ -65,4 +73,4 @@ id = "finish" kind = "donna.lib.finish" ``` -RFC work plan finalized. Ready to execute the planned workflow. +Work plan finalized. Ready to execute the planned workflow. diff --git a/donna/artifacts/rfc/work/create.md b/donna/artifacts/rfc/work/request.md similarity index 100% rename from donna/artifacts/rfc/work/create.md rename to donna/artifacts/rfc/work/request.md diff --git a/donna/artifacts/usage/artifacts.md b/donna/artifacts/usage/artifacts.md index 63afa15a..e92ae770 100644 --- a/donna/artifacts/usage/artifacts.md +++ b/donna/artifacts/usage/artifacts.md @@ -168,7 +168,11 @@ id = "operation_id" kind = "donna.lib.request_action" ``` -#### Kinds of Workflow Operations +The title of the workflow section MUST be a short human-readable description of the operation in the form of an imperative verb phrase, for example, `Implement the feature X`, `Create a document Y`. + +#### Kind: Operation + +The title of the operation section MUST be a short human-readable description of the operation in the form of an imperative verb phrase, for example, `Run tests`, `Format the codebase`, `Implement function X in the module Y`.x ##### `donna.lib.request_action` diff --git a/donna/artifacts/usage/cli.md b/donna/artifacts/usage/cli.md index 6d717759..85104a8d 100644 --- a/donna/artifacts/usage/cli.md +++ b/donna/artifacts/usage/cli.md @@ -71,11 +71,12 @@ Donna renders cells differently, depending on the protocol used. ### Commands -There are three sets of commands: +There are four sets of commands: - `donna -p <protocol> workspaces …` — manages workspaces. Most-likely it will be used once per your project to initialize it. - `donna -p <protocol> sessions …` — manages sessions. You will use these commands to start, push forward, and manage your work. - `donna -p <protocol> artifacts …` — manages artifacts. You will use these commands to read and update artifacts you are working with. +- `donna -p <protocol> journal …` — manages session actions journal. You will use these commands to log and inspect the history of actions performed during the session. Use: @@ -149,10 +150,9 @@ Use the next commands to work with artifacts: - `donna -p <protocol> artifacts copy <artifact-id-from> <artifact-id-to>` — copy an artifact source to another artifact ID (can be in a different world). This overwrites the destination if it exists. - `donna -p <protocol> artifacts move <artifact-id-from> <artifact-id-to>` — copy an artifact source to another artifact ID and remove the original. This overwrites the destination if it exists. - `donna -p <protocol> artifacts remove <artifact-pattern>` — remove artifacts matching a pattern. Use this command when you need to delete artifacts. -- `donna -p <protocol> artifacts validate <world>:<artifact>` — validate the given artifact to ensure it is correct and has no issues. -- `donna -p <protocol> artifacts validate-all [<artifact-pattern>]` — validate all artifacts corresponding to the given pattern. If `<artifact-pattern>` is omitted, validate all artifacts in all worlds. +- `donna -p <protocol> artifacts validate [<artifact-pattern>]` — validate all artifacts corresponding to the given pattern. If `<artifact-pattern>` is omitted, validate all artifacts in all worlds. -Commands that accept an artifact pattern (`artifacts list`, `artifacts view`, `artifacts remove`, `artifacts validate-all`) also accept a repeatable `--tag <tag>` option to filter by artifact tags. When multiple tags are provided, only artifacts that include **all** specified tags are matched. +Commands that accept an artifact pattern (`artifacts list`, `artifacts view`, `artifacts remove`, `artifacts validate`) also accept a repeatable `--tag <tag>` option to filter by artifact tags. When multiple tags are provided, only artifacts that include **all** specified tags are matched. The format of `<artifact-pattern>` is as follows: @@ -165,6 +165,43 @@ The format of `<artifact-pattern>` is as follows: - `world:**` — matches all artifacts in the `world` world. - `world:**:name` — matches all artifacts with id ending with `:name` in the `world` world. +### Working with journal + +Use the next commands to work with session journal: + +- `donna -p <protocol> journal write <message>` — record a single new entry to the journal with the given **single-line** `message` (newlines are not allowed). Donna automatically adds a timestamp and other relevant information to the journal entry. +- `donna -p <protocol> journal view [--lines N] [--follow]` — display journal records. + +Agents MUST use `donna -p <protocol> journal write <message>` to log: + +- Goals of the long-running agent-side operations: `Goal: <goal description>`. +- Significant steps of the long-running agent-side operations: `Step: <phase progress or completion handoff>`. +- Significant thoughts during the long-running operations: `Thought: <important thought>`. +- Significant assumptions during the long-running operations: `Assumption: <important assumption>`. +- Changes in the project source code or in the project structure: `Change: <what changed and where>`. + +For each non-trivial action request, agents MUST follow this journaling contract: + +1. Write exactly one `Goal:` record at action-request start. +2. Write `Step:` records at significant phase boundaries. If an action request describes a multi-step process, there MUST be at least one `Step:` record per specified step and one `Step:` record for the completion handoff. +3. Write `Change:` records after each meaningful source update batch. +4. Write one final `Step:` record immediately before `sessions action-request-completed`. + +Agents MUST consider these cases as significant phase boundaries: + +- A work phase expected to take more than 10 seconds. +- Transition from analysis/research to implementation/editing. +- Transition to a new step in a multi-step process described in the action request. +- Start or completion of a multi-file or multi-artifact change batch. +- A decision that changes implementation direction. + +Before `sessions action-request-completed`, agents MUST check journal completeness for the current action request. + +Agents MUST NOT log: + +- CLI commands they execute. +- Elementary/trivial steps. + ## IMPORTANT ON DONNA TOOL USAGE **Strictly follow described command syntax** diff --git a/donna/cli/__main__.py b/donna/cli/__main__.py index a7d69b0d..07c44647 100644 --- a/donna/cli/__main__.py +++ b/donna/cli/__main__.py @@ -1,5 +1,6 @@ from donna.cli.application import app # noqa: F401 from donna.cli.commands import artifacts # noqa: F401 +from donna.cli.commands import journal # noqa: F401 from donna.cli.commands import sessions # noqa: F401 from donna.cli.commands import workspaces # noqa: F401 diff --git a/donna/cli/commands/artifacts.py b/donna/cli/commands/artifacts.py index 1c10d340..ac5bccff 100644 --- a/donna/cli/commands/artifacts.py +++ b/donna/cli/commands/artifacts.py @@ -19,6 +19,8 @@ from donna.core.errors import ErrorsList from donna.core.result import Err, Ok, Result from donna.domain.ids import FullArtifactIdPattern +from donna.machine import journal as machine_journal +from donna.machine.sessions import load_state from donna.protocol.cell_shortcuts import operation_succeeded from donna.protocol.cells import Cell from donna.workspaces import artifacts as world_artifacts @@ -41,6 +43,32 @@ def _parse_slug_with_extension(value: str) -> Result[tuple[str, str], ErrorsList return Ok((slug, extension)) +def _log_artifact_operation(message: str) -> None: + state_result = load_state() + + if state_result.is_err(): + # log nothing if we have no session state + return + + state = state_result.unwrap() + + machine_journal.add( + message=message, + current_task_id=str(state.current_task.id) if state.current_task else None, + current_work_unit_id=None, + current_operation_id=None, + ) + + +def _log_operation_on_artifacts(message: str, pattern: FullArtifactIdPattern, tags: TagOption | None) -> None: + if not tags: + return _log_artifact_operation(f"{message} `{pattern}`") + + tags_list = ", ".join(f"'{tag}'" for tag in tags) + + return _log_artifact_operation(f"{message} `{pattern}` with tags {tags_list}") + + @artifacts_cli.command( help="List artifacts matching a pattern and show their status summaries. Lists all all artifacts by default." ) @@ -49,6 +77,8 @@ def list( pattern: FullArtifactIdPatternArgument = DEFAULT_ARTIFACT_PATTERN, tags: TagOption = None, ) -> Iterable[Cell]: + _log_operation_on_artifacts("List artifacts", pattern, tags) + artifacts = world_artifacts.list_artifacts(pattern, tags=tags).unwrap() return [artifact.node().status() for artifact in artifacts] @@ -60,6 +90,8 @@ def view( pattern: FullArtifactIdPatternArgument, tags: TagOption = None, ) -> Iterable[Cell]: + _log_operation_on_artifacts("View artifacts", pattern, tags) + artifacts = world_artifacts.list_artifacts(pattern, tags=tags).unwrap() return [artifact.node().info() for artifact in artifacts] @@ -76,6 +108,8 @@ def fetch(id: FullArtifactIdArgument, output: OutputPathOption = None) -> Iterab extension = world_artifacts.artifact_file_extension(id).unwrap() output = world_tmp.file_for_artifact(id, extension) + _log_artifact_operation(f"Fetch artifact `{id}` to '{output}'") + world_artifacts.fetch_artifact(id, output).unwrap() return [ @@ -93,6 +127,8 @@ def tmp( slug, extension = _parse_slug_with_extension(slug_with_extension).unwrap() output = world_tmp.create_file_for_slug(slug, extension) + _log_artifact_operation(f"Created temporary file {output}") + return [ operation_succeeded( f"Temporary file created at '{output}'", @@ -117,6 +153,8 @@ def update( input_path = input input_display = str(input) + _log_artifact_operation(f"Update artifact `{id}` from '{input_display}'") + world_artifacts.update_artifact(id, input_path, extension=extension).unwrap() return [ operation_succeeded( @@ -130,6 +168,8 @@ def update( @artifacts_cli.command(help="Copy an artifact to another artifact ID (possibly across worlds).") @cells_cli def copy(source_id: FullArtifactIdArgument, target_id: FullArtifactIdArgument) -> Iterable[Cell]: + _log_artifact_operation(f"Copy artifact from `{source_id}` to `{target_id}`") + world_artifacts.copy_artifact(source_id, target_id).unwrap() return [ operation_succeeded( @@ -143,6 +183,8 @@ def copy(source_id: FullArtifactIdArgument, target_id: FullArtifactIdArgument) - @artifacts_cli.command(help="Move an artifact to another artifact ID (possibly across worlds).") @cells_cli def move(source_id: FullArtifactIdArgument, target_id: FullArtifactIdArgument) -> Iterable[Cell]: + _log_artifact_operation(f"Move artifact from `{source_id}` to `{target_id}`") + world_artifacts.move_artifact(source_id, target_id).unwrap() return [ operation_succeeded( @@ -159,6 +201,8 @@ def remove( pattern: FullArtifactIdPatternArgument, tags: TagOption = None, ) -> Iterable[Cell]: + _log_operation_on_artifacts("Remove artifacts", pattern, tags) + artifacts = world_artifacts.list_artifacts(pattern, tags=tags).unwrap() cells: builtins.list[Cell] = [] @@ -169,24 +213,14 @@ def remove( return cells -@artifacts_cli.command(help="Validate an artifact and return any validation errors.") +@artifacts_cli.command(help="Validate artifacts matching a pattern (defaults to all artifacts) and return any errors.") @cells_cli -def validate(id: FullArtifactIdArgument) -> Iterable[Cell]: - artifact = world_artifacts.load_artifact(id).unwrap() - - artifact.validate_artifact().unwrap() - - return [operation_succeeded(f"Artifact `{id}` is valid", artifact_id=str(id))] - - -@artifacts_cli.command( - help="Validate all artifacts matching a pattern (defaults to all artifacts) and return any errors." -) -@cells_cli -def validate_all( +def validate( pattern: FullArtifactIdPatternArgument = DEFAULT_ARTIFACT_PATTERN, tags: TagOption = None, ) -> Iterable[Cell]: # noqa: CCR001 + _log_operation_on_artifacts("Validate artifacts", pattern, tags) + artifacts = world_artifacts.list_artifacts(pattern, tags=tags).unwrap() errors = [] diff --git a/donna/cli/commands/journal.py b/donna/cli/commands/journal.py new file mode 100644 index 00000000..3a14cc56 --- /dev/null +++ b/donna/cli/commands/journal.py @@ -0,0 +1,57 @@ +import sys +from collections.abc import Iterable + +import typer + +from donna.cli.application import app +from donna.cli.utils import cells_cli, output_cells +from donna.machine import journal as machine_journal +from donna.machine.sessions import load_state +from donna.protocol.cell_shortcuts import operation_succeeded +from donna.protocol.cells import Cell +from donna.protocol.modes import get_cell_formatter + +journal_cli = typer.Typer() + + +@journal_cli.command(help="Append a new journal record.") +@cells_cli +def write( + message: str = typer.Argument(..., help="Single-line message to append to journal (newlines are not allowed)."), +) -> Iterable[Cell]: + state = load_state().unwrap() + + machine_journal.add( + message=message, + current_task_id=str(state.current_task.id) if state.current_task else None, + current_work_unit_id=None, + current_operation_id=None, + ).unwrap() + return [operation_succeeded("Journal record appended.")] + + +@journal_cli.command(help="View journal records.") +def view( # noqa: CCR001 + lines: int | None = typer.Option(None, min=1, help="Show only the last N records."), + follow: bool = typer.Option(False, help="Keep printing records as they are appended."), +) -> None: + iterator = machine_journal.read(lines=lines, follow=follow) + + formatter = get_cell_formatter() + + for record_result in iterator: + if record_result.is_err(): + output_cells([error.node().info() for error in record_result.unwrap_err()]) + return + + record = record_result.unwrap() + rendered = formatter.format_journal(record) + sys.stdout.buffer.write(rendered + b"\n") + sys.stdout.buffer.flush() + + +app.add_typer( + journal_cli, + name="journal", + help="Append and inspect session actions journal records.", +) diff --git a/donna/cli/commands/sessions.py b/donna/cli/commands/sessions.py index b5d1015b..f765e8bf 100644 --- a/donna/cli/commands/sessions.py +++ b/donna/cli/commands/sessions.py @@ -14,13 +14,13 @@ @sessions_cli.command(help="Start a new session, reset session state, remove all session artifacts.") @cells_cli def start() -> Iterable[Cell]: - return sessions.start() + return sessions.start().unwrap() @sessions_cli.command(help="Reset the current session state, keeps session artifacts.") @cells_cli def reset() -> Iterable[Cell]: - return sessions.reset() + return sessions.reset().unwrap() @sessions_cli.command( @@ -29,25 +29,25 @@ def reset() -> Iterable[Cell]: ) @cells_cli def continue_() -> Iterable[Cell]: - return sessions.continue_() + return sessions.continue_().unwrap() @sessions_cli.command(help="Show a concise status summary for the current session, including pending action requests.") @cells_cli def status() -> Iterable[Cell]: - return sessions.status() + return sessions.status().unwrap() @sessions_cli.command(help="Show detailed session state, including action requests.") @cells_cli def details() -> Iterable[Cell]: - return sessions.details() + return sessions.details().unwrap() @sessions_cli.command(help="Run a workflow from an artifact to drive the current session forward.") @cells_cli def run(workflow_id: FullArtifactIdArgument) -> Iterable[Cell]: - return sessions.start_workflow(workflow_id) + return sessions.start_workflow(workflow_id).unwrap() @sessions_cli.command( @@ -57,7 +57,7 @@ def run(workflow_id: FullArtifactIdArgument) -> Iterable[Cell]: def action_request_completed( request_id: ActionRequestIdArgument, next_operation_id: FullArtifactSectionIdArgument ) -> Iterable[Cell]: - return sessions.complete_action_request(request_id, next_operation_id) + return sessions.complete_action_request(request_id, next_operation_id).unwrap() app.add_typer( diff --git a/donna/cli/utils.py b/donna/cli/utils.py index 71f8034f..5acea8a4 100644 --- a/donna/cli/utils.py +++ b/donna/cli/utils.py @@ -6,7 +6,7 @@ import typer -from donna.core.errors import EnvironmentError +from donna.core.errors import EnvironmentError, ErrorsList from donna.core.result import UnwrapError from donna.protocol.cells import Cell from donna.protocol.modes import Mode, get_cell_formatter @@ -17,28 +17,54 @@ def output_cells(cells: Iterable[Cell]) -> None: formatter = get_cell_formatter() - output = formatter.format_cells(list(cells)) - - sys.stdout.buffer.write(output) + for cell in cells: + output = formatter.format_cell(cell) + sys.stdout.buffer.write(output) P = ParamSpec("P") -def cells_cli(func: Callable[P, Iterable[Cell]]) -> Callable[P, None]: +def _write_errors_to_journal(errors: ErrorsList) -> None: + from donna.machine import journal as machine_journal + from donna.machine import sessions as machine_sessions + + state_result = machine_sessions.load_state() + current_task_id = None + if state_result.is_ok(): + state = state_result.unwrap() + current_task_id = str(state.current_task.id) if state.current_task else None + + for error in errors: + message = f"Error: {error.node().journal_message()} [{error.code}]" + + machine_journal.add( + message=message, + current_task_id=current_task_id, + current_work_unit_id=None, + current_operation_id=None, + actor_id="donna", + ) + + +def cells_cli(func: Callable[P, Iterable[Cell]]) -> Callable[P, None]: # noqa: CCR001 @functools.wraps(func) - def wrapper(*args: P.args, **kwargs: P.kwargs) -> None: + def wrapper(*args: P.args, **kwargs: P.kwargs) -> None: # noqa: CCR001 try: cells = func(*args, **kwargs) except UnwrapError as e: + errors: ErrorsList if isinstance(e.arguments["error"], EnvironmentError): - cells = [e.arguments["error"].node().info()] + errors = [e.arguments["error"]] elif isinstance(e.arguments["error"], Iterable): - cells = [error.node().info() for error in e.arguments["error"] if isinstance(error, EnvironmentError)] + errors = [error for error in e.arguments["error"] if isinstance(error, EnvironmentError)] else: raise + _write_errors_to_journal(errors) + cells = [error.node().info() for error in errors] + output_cells(cells) return wrapper diff --git a/donna/core/errors.py b/donna/core/errors.py index 420a2e13..667e5a18 100644 --- a/donna/core/errors.py +++ b/donna/core/errors.py @@ -112,5 +112,8 @@ def status(self) -> Cell: **self.meta(), ) + def journal_message(self) -> str: + return self._error.message.format(error=self._error).replace("\n", " ").strip() + ErrorsList = list[EnvironmentError] diff --git a/donna/core/utils.py b/donna/core/utils.py index ffb8470c..d76800d8 100644 --- a/donna/core/utils.py +++ b/donna/core/utils.py @@ -1,9 +1,14 @@ +import datetime import pathlib from donna.core import errors as core_errors from donna.core.result import Err, Ok, Result +def now() -> datetime.datetime: + return datetime.datetime.now(datetime.UTC) + + def first_donna_dir(donna_dir_name: str) -> pathlib.Path | None: """Get the first parent directory containing the donna directory. diff --git a/donna/domain/ids.py b/donna/domain/ids.py index 9ccfeeb6..f3451fdd 100644 --- a/donna/domain/ids.py +++ b/donna/domain/ids.py @@ -113,6 +113,10 @@ def validate(cls, id: str) -> bool: return crc == expected_crc + @property + def short(self) -> str: + return self.split("-")[1] + @classmethod def __get_pydantic_core_schema__(cls, source_type: Any, handler: Any) -> core_schema.CoreSchema: # noqa: CCR001 @@ -463,6 +467,12 @@ def full_artifact_id(self) -> FullArtifactId: def local_id(self) -> ArtifactSectionId: return ArtifactSectionId(self.parts[-1]) + @property + def short(self) -> str: + parts = str(self).split(self.delimiter) + new_parts = [part[0] for part in parts[:-2]] + parts[-2:] + return self.delimiter.join(new_parts) + @classmethod def parse(cls, text: str) -> Result["FullArtifactSectionId", ErrorsList]: # noqa: CCR001 if not isinstance(text, str) or not text: diff --git a/donna/machine/action_requests.py b/donna/machine/action_requests.py index 8c055c63..53bddd73 100644 --- a/donna/machine/action_requests.py +++ b/donna/machine/action_requests.py @@ -10,13 +10,15 @@ class ActionRequest(BaseEntity): id: ActionRequestId | None request: str operation_id: FullArtifactSectionId + title: str = "unknown" # TODO: remove default value after 2026.05.01 @classmethod - def build(cls, request: str, operation_id: FullArtifactSectionId) -> "ActionRequest": + def build(cls, title: str, request: str, operation_id: FullArtifactSectionId) -> "ActionRequest": return cls( id=None, request=request, operation_id=operation_id, + title=title, ) def node(self) -> "ActionRequestNode": diff --git a/donna/machine/artifacts.py b/donna/machine/artifacts.py index 81157400..919c2aca 100644 --- a/donna/machine/artifacts.py +++ b/donna/machine/artifacts.py @@ -1,4 +1,4 @@ -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, Union import pydantic @@ -191,7 +191,7 @@ def status(self) -> Cell: @unwrap_to_error def resolve( - target_id: FullArtifactSectionId, render_context: "ArtifactRenderContext" + target_id: FullArtifactSectionId, render_context: Union["ArtifactRenderContext", None] = None ) -> Result[ArtifactSection, ErrorsList]: from donna.workspaces import artifacts as world_artifacts diff --git a/donna/machine/changes.py b/donna/machine/changes.py index 6b598d4b..f0acbbd0 100644 --- a/donna/machine/changes.py +++ b/donna/machine/changes.py @@ -35,7 +35,10 @@ class ChangeAddTask(Change): operation_id: FullArtifactSectionId def apply_to(self, state: "MutableState") -> None: - task = Task.build(state.next_task_id()) + task = Task.build( + state.next_task_id(), + workflow_id=self.operation_id, + ) state.add_task(task) diff --git a/donna/machine/errors.py b/donna/machine/errors.py index 51577f77..c1ade09a 100644 --- a/donna/machine/errors.py +++ b/donna/machine/errors.py @@ -16,6 +16,10 @@ class SessionStateStatusInvalid(InternalError): message: str = "Session state status is invalid." +class UnsupportedFormatterMode(InternalError): + message: str = "Formatter for mode '{mode}' is not implemented." + + class EnvironmentError(core_errors.EnvironmentError): """Base class for environment errors in donna.machine.""" @@ -28,6 +32,15 @@ class SessionStateNotInitialized(EnvironmentError): ways_to_fix: list[str] = ["Run Donna session start to initialize session state."] +class JournalMessageContainsNewlines(EnvironmentError): + code: str = "donna.machine.journal_message_contains_newlines" + message: str = "Journal message must be a single line and must not contain newline characters." + ways_to_fix: list[str] = [ + "Provide `journal write` message as a single line.", + "Replace newline characters with spaces or split the text into multiple journal records.", + ] + + class ActionRequestNotFound(EnvironmentError): code: str = "donna.machine.action_request_not_found" message: str = "Action request `{error.request_id}` was not found in the current session state." diff --git a/donna/machine/journal.py b/donna/machine/journal.py new file mode 100644 index 00000000..31f09f6d --- /dev/null +++ b/donna/machine/journal.py @@ -0,0 +1,124 @@ +import datetime +import json +from collections.abc import Iterable +from typing import Any + +import pydantic + +from donna.core.entities import BaseEntity +from donna.core.errors import ErrorsList +from donna.core.result import Err, Ok, Result, unwrap_to_error +from donna.core.utils import now +from donna.domain.ids import FullArtifactSectionId, TaskId, WorkUnitId +from donna.machine import errors as machine_errors +from donna.workspaces import utils as workspace_utils +from donna.workspaces.config import protocol as protocol_mode + + +def message_has_newlines(message: str) -> bool: + return "\n" in message or "\r" in message + + +class JournalRecord(BaseEntity): + timestamp: datetime.datetime + actor_id: str | None + message: str + current_task_id: TaskId | None + current_work_unit_id: WorkUnitId | None + current_operation_id: FullArtifactSectionId | None + + @pydantic.field_validator("message", mode="after") + @classmethod + def validate_message_no_newlines(cls, value: str) -> str: + if message_has_newlines(value): + raise ValueError("Journal message must not contain newline characters.") + + return value + + +def serialize_record(record: JournalRecord) -> bytes: + return json.dumps( + record.model_dump(mode="json"), + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + + +def deserialize_record(content: bytes) -> JournalRecord: + payload = json.loads(content.decode("utf-8").strip()) + return JournalRecord.model_validate(payload) + + +@unwrap_to_error +def reset() -> Result[None, ErrorsList]: + workspace_utils.session_world().unwrap().journal_reset().unwrap() + return Ok(None) + + +def smart_agent_id() -> str: + from donna.protocol.modes import Mode as ProtocolMode + + match protocol_mode(): + case ProtocolMode.human: + return "human" + case ProtocolMode.llm: + return "agent" + case ProtocolMode.automation: + return "automation" + case _: + raise machine_errors.UnsupportedFormatterMode(mode=protocol_mode()) + + +@unwrap_to_error +def add( + message: str, + current_task_id: str | None, + current_work_unit_id: str | None, + current_operation_id: str | None, + **kwargs: Any, +) -> Result[JournalRecord, ErrorsList]: + from donna.protocol.utils import instant_output_journal + + if message_has_newlines(message): + return Err([machine_errors.JournalMessageContainsNewlines()]) + + if "actor_id" in kwargs: + actor_id = kwargs["actor_id"] + else: + actor_id = smart_agent_id() + + parsed_task_id = TaskId(current_task_id) if current_task_id is not None else None + parsed_work_unit_id = WorkUnitId(current_work_unit_id) if current_work_unit_id is not None else None + parsed_operation_id = FullArtifactSectionId(current_operation_id) if current_operation_id is not None else None + + record = JournalRecord( + timestamp=now(), + actor_id=actor_id, + message=message, + current_task_id=parsed_task_id, + current_work_unit_id=parsed_work_unit_id, + current_operation_id=parsed_operation_id, + ) + + serialized = serialize_record(record) + workspace_utils.session_world().unwrap().journal_add(serialized).unwrap() + + instant_output_journal(record) + + return Ok(record) + + +def read(lines: int | None = None, follow: bool = False) -> Iterable[Result[JournalRecord, ErrorsList]]: + session_world_result = workspace_utils.session_world() + if session_world_result.is_err(): + yield Err(session_world_result.unwrap_err()) + return + + raw_records = session_world_result.unwrap().journal_read(lines=lines, follow=follow) + for raw_record_result in raw_records: + if raw_record_result.is_err(): + yield Err(raw_record_result.unwrap_err()) + continue + + yield Ok(deserialize_record(raw_record_result.unwrap())) diff --git a/donna/machine/primitives.py b/donna/machine/primitives.py index a53d5a3a..a6efb0e2 100644 --- a/donna/machine/primitives.py +++ b/donna/machine/primitives.py @@ -1,5 +1,5 @@ import importlib -from typing import TYPE_CHECKING, Any, ClassVar, Iterable +from typing import TYPE_CHECKING, Any, ClassVar from jinja2.runtime import Context @@ -28,7 +28,9 @@ class Primitive(BaseEntity): def validate_section(self, artifact: "Artifact", section_id: ArtifactSectionId) -> Result[None, ErrorsList]: return Ok(None) - def execute_section(self, task: "Task", unit: "WorkUnit", section: "ArtifactSection") -> Iterable["Change"]: + def execute_section( + self, task: "Task", unit: "WorkUnit", section: "ArtifactSection" + ) -> Result[list["Change"], ErrorsList]: raise machine_errors.PrimitiveMethodUnsupported( primitive_name=self.__class__.__name__, method_name="execute_section()" ) diff --git a/donna/machine/sessions.py b/donna/machine/sessions.py index c2d75c85..eef8de8f 100644 --- a/donna/machine/sessions.py +++ b/donna/machine/sessions.py @@ -3,35 +3,21 @@ from donna.core.errors import ErrorsList from donna.core.result import Err, Ok, Result, unwrap_to_error -from donna.domain.ids import ActionRequestId, FullArtifactId, FullArtifactSectionId, WorldId +from donna.domain.ids import ActionRequestId, FullArtifactId, FullArtifactSectionId from donna.machine import errors as machine_errors +from donna.machine import journal as machine_journal from donna.machine.operations import OperationMeta from donna.machine.state import ConsistentState, MutableState from donna.protocol.cell_shortcuts import operation_succeeded from donna.protocol.cells import Cell from donna.workspaces import artifacts from donna.workspaces import tmp as world_tmp -from donna.workspaces.config import config -from donna.workspaces.worlds.base import World - - -def _errors_to_cells(errors: ErrorsList) -> list[Cell]: - return [error.node().info() for error in errors] - - -@unwrap_to_error -def _session() -> Result[World, ErrorsList]: - world = config().get_world(WorldId("session")).unwrap() - - if not world.is_initialized(): - world.initialize(reset=False) - - return Ok(world) +from donna.workspaces import utils as workspace_utils @unwrap_to_error -def _load_state() -> Result[ConsistentState, ErrorsList]: - content = _session().unwrap().read_state("state.json").unwrap() +def load_state() -> Result[ConsistentState, ErrorsList]: + content = workspace_utils.session_world().unwrap().read_state("state.json").unwrap() if content is None: return Err([machine_errors.SessionStateNotInitialized()]) @@ -40,7 +26,7 @@ def _load_state() -> Result[ConsistentState, ErrorsList]: @unwrap_to_error def _save_state(state: ConsistentState) -> Result[None, ErrorsList]: - _session().unwrap().write_state("state.json", state.to_json().encode("utf-8")).unwrap() + workspace_utils.session_world().unwrap().write_state("state.json", state.to_json().encode("utf-8")).unwrap() return Ok(None) @@ -55,127 +41,89 @@ def _state_run(mutator: MutableState) -> Result[None, ErrorsList]: @unwrap_to_error def _state_cells() -> Result[list[Cell], ErrorsList]: - return Ok(_load_state().unwrap().node().details()) + return Ok(load_state().unwrap().node().details()) P = ParamSpec("P") +CellsResult = Result[list[Cell], ErrorsList] -def _session_required(func: Callable[P, list[Cell]]) -> Callable[P, list[Cell]]: +def _session_required( + func: Callable[P, CellsResult], +) -> Callable[P, CellsResult]: # TODO: refactor to catch domain exception from load_state # when we implement domain exceptions @functools.wraps(func) - def wrapper(*args: P.args, **kwargs: P.kwargs) -> list[Cell]: - state_result = _load_state() - if state_result.is_err(): - return _errors_to_cells(state_result.unwrap_err()) - + @unwrap_to_error + def wrapper(*args: P.args, **kwargs: P.kwargs) -> CellsResult: + load_state().unwrap() return func(*args, **kwargs) return wrapper -def start() -> list[Cell]: +@unwrap_to_error +def start() -> Result[list[Cell], ErrorsList]: world_tmp.clear() - session_result = _session() - if session_result.is_err(): - return _errors_to_cells(session_result.unwrap_err()) - - session_result.unwrap().initialize(reset=True) + workspace_utils.session_world().unwrap().initialize(reset=True) - save_result = _save_state(MutableState.build().freeze()) - if save_result.is_err(): - return _errors_to_cells(save_result.unwrap_err()) + machine_journal.reset().unwrap() - return [operation_succeeded("Started new session.")] + machine_journal.add( + message="Started new session.", + current_task_id=None, + current_work_unit_id=None, + current_operation_id=None, + ).unwrap() + _save_state(MutableState.build().freeze()).unwrap() + return Ok([operation_succeeded("Started new session.")]) -def reset() -> list[Cell]: - save_result = _save_state(MutableState.build().freeze()) - if save_result.is_err(): - return _errors_to_cells(save_result.unwrap_err()) - return [operation_succeeded("Session state reset.")] +@unwrap_to_error +def reset() -> Result[list[Cell], ErrorsList]: + _save_state(MutableState.build().freeze()).unwrap() + return Ok([operation_succeeded("Session state reset.")]) -def clear() -> list[Cell]: +@unwrap_to_error +def clear() -> Result[list[Cell], ErrorsList]: world_tmp.clear() - session_result = _session() - if session_result.is_err(): - return _errors_to_cells(session_result.unwrap_err()) - - session_result.unwrap().initialize(reset=True) - return [operation_succeeded("Cleared session.")] + workspace_utils.session_world().unwrap().initialize(reset=True) + return Ok([operation_succeeded("Cleared session.")]) @_session_required -def continue_() -> list[Cell]: - state_result = _load_state() - if state_result.is_err(): - return _errors_to_cells(state_result.unwrap_err()) - - mutator = state_result.unwrap().mutator() - run_result = _state_run(mutator) - if run_result.is_err(): - return _errors_to_cells(run_result.unwrap_err()) - - cells_result = _state_cells() - if cells_result.is_err(): - return _errors_to_cells(cells_result.unwrap_err()) - - return cells_result.unwrap() +@unwrap_to_error +def continue_() -> Result[list[Cell], ErrorsList]: + mutator = load_state().unwrap().mutator() + _state_run(mutator).unwrap() + return _state_cells() @_session_required -def status() -> list[Cell]: - state_result = _load_state() - if state_result.is_err(): - return _errors_to_cells(state_result.unwrap_err()) - - return [state_result.unwrap().node().info()] +@unwrap_to_error +def status() -> Result[list[Cell], ErrorsList]: + return Ok([load_state().unwrap().node().info()]) @_session_required -def details() -> list[Cell]: - state_result = _load_state() - if state_result.is_err(): - return _errors_to_cells(state_result.unwrap_err()) - - return state_result.unwrap().node().details() +@unwrap_to_error +def details() -> Result[list[Cell], ErrorsList]: + return Ok(load_state().unwrap().node().details()) @_session_required -def start_workflow(artifact_id: FullArtifactId) -> list[Cell]: - workflow_result = artifacts.load_artifact(artifact_id) - if workflow_result.is_err(): - return _errors_to_cells(workflow_result.unwrap_err()) - - workflow = workflow_result.unwrap() - primary_section_result = workflow.primary_section() - if primary_section_result.is_err(): - return _errors_to_cells(primary_section_result.unwrap_err()) - - primary_section = primary_section_result.unwrap() - - state_result = _load_state() - if state_result.is_err(): - return _errors_to_cells(state_result.unwrap_err()) - - mutator = state_result.unwrap().mutator() - mutator.start_workflow(workflow.id.to_full_local(primary_section.id)) - save_result = _save_state(mutator.freeze()) - if save_result.is_err(): - return _errors_to_cells(save_result.unwrap_err()) - - run_result = _state_run(mutator) - if run_result.is_err(): - return _errors_to_cells(run_result.unwrap_err()) - - cells_result = _state_cells() - if cells_result.is_err(): - return _errors_to_cells(cells_result.unwrap_err()) - - return cells_result.unwrap() +@unwrap_to_error +def start_workflow(artifact_id: FullArtifactId) -> Result[list[Cell], ErrorsList]: # noqa: CCR001 + static_state = load_state().unwrap() + workflow = artifacts.load_artifact(artifact_id).unwrap() + primary_section = workflow.primary_section().unwrap() + mutator = static_state.mutator() + mutator.start_workflow(workflow.id.to_full_local(primary_section.id)).unwrap() + _save_state(mutator.freeze()).unwrap() + _state_run(mutator).unwrap() + return _state_cells() @unwrap_to_error @@ -197,27 +145,13 @@ def _validate_operation_transition( @_session_required -def complete_action_request(request_id: ActionRequestId, next_operation_id: FullArtifactSectionId) -> list[Cell]: - state_result = _load_state() - if state_result.is_err(): - return _errors_to_cells(state_result.unwrap_err()) - - mutator = state_result.unwrap().mutator() - transition_result = _validate_operation_transition(mutator, request_id, next_operation_id) - if transition_result.is_err(): - return _errors_to_cells(transition_result.unwrap_err()) - - mutator.complete_action_request(request_id, next_operation_id) - save_result = _save_state(mutator.freeze()) - if save_result.is_err(): - return _errors_to_cells(save_result.unwrap_err()) - - run_result = _state_run(mutator) - if run_result.is_err(): - return _errors_to_cells(run_result.unwrap_err()) - - cells_result = _state_cells() - if cells_result.is_err(): - return _errors_to_cells(cells_result.unwrap_err()) - - return cells_result.unwrap() +@unwrap_to_error +def complete_action_request( + request_id: ActionRequestId, next_operation_id: FullArtifactSectionId +) -> Result[list[Cell], ErrorsList]: + mutator = load_state().unwrap().mutator() + _validate_operation_transition(mutator, request_id, next_operation_id).unwrap() + mutator.complete_action_request(request_id, next_operation_id).unwrap() + _save_state(mutator.freeze()).unwrap() + _state_run(mutator).unwrap() + return _state_cells() diff --git a/donna/machine/state.py b/donna/machine/state.py index 14f552f1..6b6e47cc 100644 --- a/donna/machine/state.py +++ b/donna/machine/state.py @@ -15,7 +15,9 @@ WorkUnitId, ) from donna.machine import errors as machine_errors +from donna.machine import journal as machine_journal from donna.machine.action_requests import ActionRequest +from donna.machine.artifacts import resolve from donna.machine.changes import ( Change, ChangeAddTask, @@ -47,7 +49,10 @@ def node(self) -> "StateNode": ########### @property - def current_task(self) -> Task: + def current_task(self) -> Task | None: + if not self.tasks: + return None + return self.tasks[-1] def get_action_request(self, request_id: ActionRequestId) -> Result[ActionRequest, ErrorsList]: @@ -61,6 +66,9 @@ def get_action_request(self, request_id: ActionRequestId) -> Result[ActionReques # Since we only append work units, this effectively works as a queue per task # In the future we may want to have more sophisticated scheduling def get_next_work_unit(self) -> WorkUnit | None: + if self.current_task is None: + return None + for work_unit in self.work_units: if work_unit.task_id != self.current_task.id: continue @@ -119,6 +127,15 @@ def mark_started(self) -> None: def add_action_request(self, action_request: ActionRequest) -> None: full_request = action_request.replace(id=self.next_action_request_id()) + + machine_journal.add( + actor_id="donna", + message=f"Request agent action `{full_request.title}`", + current_task_id=str(self.current_task.id) if self.current_task else None, + current_work_unit_id=None, + current_operation_id=None, + ).unwrap() + self.action_requests.append(full_request) def add_work_unit(self, work_unit: WorkUnit) -> None: @@ -144,18 +161,55 @@ def apply_changes(self, changes: Sequence[Change]) -> None: # Complex operations #################### - def complete_action_request(self, request_id: ActionRequestId, next_operation_id: FullArtifactSectionId) -> None: + @unwrap_to_error + def complete_action_request( + self, request_id: ActionRequestId, next_operation_id: FullArtifactSectionId + ) -> Result[None, ErrorsList]: + current_task = self.current_task + assert current_task is not None + + action_request = self.get_action_request(request_id).unwrap() + machine_journal.add( + message=f"Complete agent action `{action_request.title}`", + current_task_id=str(current_task.id), + current_work_unit_id=None, + current_operation_id=None, + ).unwrap() + changes = [ - ChangeAddWorkUnit(task_id=self.current_task.id, operation_id=next_operation_id), + ChangeAddWorkUnit(task_id=current_task.id, operation_id=next_operation_id), ChangeRemoveActionRequest(action_request_id=request_id), ] self.apply_changes(changes) + return Ok(None) + + @unwrap_to_error + def start_workflow(self, full_operation_id: FullArtifactSectionId) -> Result[None, ErrorsList]: + workflow = resolve(full_operation_id).unwrap() + + machine_journal.add( + message=f"Start workflow `{workflow.title}`", + current_task_id=str(self.current_task.id) if self.current_task else None, + current_work_unit_id=None, + current_operation_id=None, + ).unwrap() - def start_workflow(self, full_operation_id: FullArtifactSectionId) -> None: changes = [ChangeAddTask(operation_id=full_operation_id)] self.apply_changes(changes) + return Ok(None) def finish_workflow(self, task_id: TaskId) -> None: + task = self.current_task + assert task is not None + workflow = resolve(task.workflow_id).unwrap() + + machine_journal.add( + message=f"Finish workflow `{workflow.title}`", + current_task_id=str(self.current_task.id) if self.current_task else None, + current_work_unit_id=None, + current_operation_id=None, + ).unwrap() + changes = [ChangeRemoveTask(task_id=task_id)] self.apply_changes(changes) @@ -163,8 +217,10 @@ def finish_workflow(self, task_id: TaskId) -> None: def execute_next_work_unit(self) -> Result[None, ErrorsList]: next_work_unit = self.get_next_work_unit() assert next_work_unit is not None + current_task = self.current_task + assert current_task is not None - changes = next_work_unit.run(self.current_task).unwrap() + changes = next_work_unit.run(current_task).unwrap() changes.append(ChangeRemoveWorkUnit(work_unit_id=next_work_unit.id)) self.apply_changes(changes) @@ -181,7 +237,7 @@ def status(self) -> Cell: if not self._state.started: message = textwrap.dedent( """ - The session has not been started yet. You can safely start a new session and then run a workflow. + This is a new session; no tasks were performed. You can safely run a workflow. """ ) diff --git a/donna/machine/tasks.py b/donna/machine/tasks.py index c321359f..3a10d8f8 100644 --- a/donna/machine/tasks.py +++ b/donna/machine/tasks.py @@ -5,8 +5,6 @@ from donna.core.errors import ErrorsList from donna.core.result import Ok, Result, unwrap_to_error from donna.domain.ids import FullArtifactSectionId, TaskId, WorkUnitId -from donna.protocol.cells import Cell -from donna.protocol.utils import instant_output if TYPE_CHECKING: from donna.machine.changes import Change @@ -14,12 +12,14 @@ class Task(BaseEntity): id: TaskId + workflow_id: FullArtifactSectionId context: dict[str, Any] @classmethod - def build(cls, id: TaskId) -> "Task": + def build(cls, id: TaskId, workflow_id: FullArtifactSectionId) -> "Task": return Task( id=id, + workflow_id=workflow_id, context={}, ) @@ -54,6 +54,7 @@ def build( @unwrap_to_error def run(self, task: Task) -> Result[list["Change"], ErrorsList]: from donna.machine import artifacts as machine_artifacts + from donna.machine import journal as machine_journal from donna.machine.primitives import resolve_primitive from donna.workspaces.artifacts import ArtifactRenderContext from donna.workspaces.templates import RenderMode @@ -66,10 +67,14 @@ def run(self, task: Task) -> Result[list["Change"], ErrorsList]: operation = machine_artifacts.resolve(self.operation_id, render_context).unwrap() operation_kind = resolve_primitive(operation.kind).unwrap() - log_message = f"{self.operation_id}: {operation.title}" - log_cell = Cell.build(kind="donna_log", media_type="text/plain", content=log_message) - instant_output([log_cell]) + machine_journal.add( + actor_id="donna", + message=operation.title, + current_task_id=str(task.id), + current_work_unit_id=str(self.id), + current_operation_id=str(self.operation_id), + ).unwrap() - changes = list(operation_kind.execute_section(task, self, operation)) + changes = operation_kind.execute_section(task, self, operation).unwrap() return Ok(changes) diff --git a/donna/primitives/artifacts/workflow.py b/donna/primitives/artifacts/workflow.py index 6e04f2b7..7d0f3006 100644 --- a/donna/primitives/artifacts/workflow.py +++ b/donna/primitives/artifacts/workflow.py @@ -1,4 +1,4 @@ -from typing import TYPE_CHECKING, ClassVar, Iterable, cast +from typing import TYPE_CHECKING, ClassVar, cast import pydantic @@ -22,10 +22,6 @@ class InternalError(core_errors.InternalError): """Base class for internal errors in donna.primitives.artifacts.workflow.""" -class WorkflowSectionMissingMetadata(InternalError): - message: str = "Workflow section is missing workflow metadata." - - class WrongStartOperation(ArtifactValidationError): code: str = "donna.workflows.wrong_start_operation" message: str = "Can not find the start operation `{error.start_operation_id}` in the workflow." @@ -128,15 +124,18 @@ def markdown_construct_meta( workflow_config = cast(WorkflowConfig, section_config) return Ok(WorkflowMeta(start_operation_id=workflow_config.start_operation_id)) - def execute_section(self, task: "Task", unit: "WorkUnit", section: ArtifactSection) -> Iterable["Change"]: + @unwrap_to_error + def execute_section( + self, task: "Task", unit: "WorkUnit", section: ArtifactSection + ) -> Result[list["Change"], ErrorsList]: from donna.machine.changes import ChangeAddWorkUnit if not isinstance(section.meta, WorkflowMeta): - raise WorkflowSectionMissingMetadata() + return Err([WorkflowSectionNotWorkflow(artifact_id=section.artifact_id, section_id=section.id)]) full_id = section.artifact_id.to_full_local(section.meta.start_operation_id) - yield ChangeAddWorkUnit(task_id=task.id, operation_id=full_id) + return Ok([ChangeAddWorkUnit(task_id=task.id, operation_id=full_id)]) @unwrap_to_error def validate_section( # noqa: CCR001, CFQ001 diff --git a/donna/primitives/operations/finish_workflow.py b/donna/primitives/operations/finish_workflow.py index f09d1b04..fbfb21c3 100644 --- a/donna/primitives/operations/finish_workflow.py +++ b/donna/primitives/operations/finish_workflow.py @@ -1,12 +1,12 @@ -from typing import TYPE_CHECKING, ClassVar, Iterator, Literal, cast +from typing import TYPE_CHECKING, ClassVar, Literal, cast from donna.core.errors import ErrorsList -from donna.core.result import Ok, Result +from donna.core.result import Ok, Result, unwrap_to_error from donna.domain.ids import FullArtifactId from donna.machine.artifacts import ArtifactSection, ArtifactSectionConfig, ArtifactSectionMeta from donna.machine.operations import FsmMode, OperationConfig, OperationKind, OperationMeta -from donna.protocol.cells import Cell -from donna.protocol.utils import instant_output +from donna.protocol import cell_shortcuts +from donna.protocol.utils import instant_output_cell from donna.workspaces import markdown from donna.workspaces.sources.markdown import MarkdownSectionMixin @@ -20,19 +20,16 @@ class FinishWorkflowConfig(OperationConfig): class FinishWorkflow(MarkdownSectionMixin, OperationKind): - def execute_section(self, task: "Task", unit: "WorkUnit", operation: ArtifactSection) -> Iterator["Change"]: + @unwrap_to_error + def execute_section( + self, task: "Task", unit: "WorkUnit", operation: ArtifactSection + ) -> Result[list["Change"], ErrorsList]: from donna.machine.changes import ChangeFinishTask - output_text = operation.description + info = cell_shortcuts.info(operation.description) + instant_output_cell(info) - output_cell = Cell.build_markdown( - kind="operation_output", - content=output_text, - operation_id=str(unit.operation_id), - ) - instant_output([output_cell]) - - yield ChangeFinishTask(task_id=task.id) + return Ok([ChangeFinishTask(task_id=task.id)]) config_class: ClassVar[type[FinishWorkflowConfig]] = FinishWorkflowConfig diff --git a/donna/primitives/operations/output.py b/donna/primitives/operations/output.py index 08dc6a40..a99e28d1 100644 --- a/donna/primitives/operations/output.py +++ b/donna/primitives/operations/output.py @@ -1,13 +1,13 @@ -from typing import TYPE_CHECKING, ClassVar, Iterator, cast +from typing import TYPE_CHECKING, ClassVar, cast from donna.core.errors import ErrorsList -from donna.core.result import Err, Ok, Result +from donna.core.result import Err, Ok, Result, unwrap_to_error from donna.domain.ids import ArtifactSectionId, FullArtifactId from donna.machine.artifacts import Artifact, ArtifactSection, ArtifactSectionConfig, ArtifactSectionMeta from donna.machine.errors import ArtifactValidationError from donna.machine.operations import OperationConfig, OperationKind, OperationMeta -from donna.protocol.cells import Cell -from donna.protocol.utils import instant_output +from donna.protocol import cell_shortcuts +from donna.protocol.utils import instant_output_cell from donna.workspaces import markdown from donna.workspaces.sources.markdown import MarkdownSectionMixin @@ -57,25 +57,22 @@ def markdown_construct_meta( ) ) - def execute_section(self, task: "Task", unit: "WorkUnit", operation: ArtifactSection) -> Iterator["Change"]: + @unwrap_to_error + def execute_section( + self, task: "Task", unit: "WorkUnit", operation: ArtifactSection + ) -> Result[list["Change"], ErrorsList]: from donna.machine.changes import ChangeAddWorkUnit meta = cast(OutputMeta, operation.meta) - output_text = operation.description - - output_cell = Cell.build_markdown( - kind="operation_output", - content=output_text, - operation_id=str(unit.operation_id), - ) - instant_output([output_cell]) + info = cell_shortcuts.info(operation.description) + instant_output_cell(info) next_operation_id = meta.next_operation_id assert next_operation_id is not None full_operation_id = unit.operation_id.full_artifact_id.to_full_local(next_operation_id) - yield ChangeAddWorkUnit(task_id=task.id, operation_id=full_operation_id) + return Ok([ChangeAddWorkUnit(task_id=task.id, operation_id=full_operation_id)]) def validate_section(self, artifact: Artifact, section_id: ArtifactSectionId) -> Result[None, ErrorsList]: section = artifact.get_section(section_id).unwrap() diff --git a/donna/primitives/operations/request_action.py b/donna/primitives/operations/request_action.py index 2304319a..3be5c584 100644 --- a/donna/primitives/operations/request_action.py +++ b/donna/primitives/operations/request_action.py @@ -1,10 +1,10 @@ import re -from typing import TYPE_CHECKING, ClassVar, Iterator, cast +from typing import TYPE_CHECKING, ClassVar, cast import pydantic from donna.core.errors import ErrorsList -from donna.core.result import Ok, Result +from donna.core.result import Ok, Result, unwrap_to_error from donna.domain import errors as domain_errors from donna.domain.ids import ArtifactSectionId, FullArtifactId from donna.machine.action_requests import ActionRequest @@ -71,13 +71,16 @@ def markdown_construct_meta( ) ) - def execute_section(self, task: "Task", unit: "WorkUnit", operation: ArtifactSection) -> Iterator["Change"]: + @unwrap_to_error + def execute_section( + self, task: "Task", unit: "WorkUnit", operation: ArtifactSection + ) -> Result[list["Change"], ErrorsList]: from donna.machine.changes import ChangeAddActionRequest request_text = operation.description full_operation_id = unit.operation_id - request = ActionRequest.build(request_text, full_operation_id) + request = ActionRequest.build(operation.title, request_text, full_operation_id) - yield ChangeAddActionRequest(action_request=request) + return Ok([ChangeAddActionRequest(action_request=request)]) diff --git a/donna/primitives/operations/run_script.py b/donna/primitives/operations/run_script.py index 2a385cce..c856a71e 100644 --- a/donna/primitives/operations/run_script.py +++ b/donna/primitives/operations/run_script.py @@ -2,14 +2,15 @@ import pathlib import subprocess # noqa: S404 import tempfile -from typing import TYPE_CHECKING, ClassVar, Iterator, cast +from typing import TYPE_CHECKING, ClassVar, cast import pydantic from donna.core import errors as core_errors from donna.core.errors import ErrorsList -from donna.core.result import Err, Ok, Result +from donna.core.result import Err, Ok, Result, unwrap_to_error from donna.domain.ids import ArtifactSectionId, FullArtifactId +from donna.machine import journal as machine_journal from donna.machine.artifacts import Artifact, ArtifactSection, ArtifactSectionConfig, ArtifactSectionMeta from donna.machine.errors import ArtifactValidationError from donna.machine.operations import OperationConfig, OperationKind, OperationMeta @@ -149,7 +150,10 @@ def markdown_construct_meta( ) ) - def execute_section(self, task: "Task", unit: "WorkUnit", operation: ArtifactSection) -> Iterator["Change"]: + @unwrap_to_error + def execute_section( + self, task: "Task", unit: "WorkUnit", operation: ArtifactSection + ) -> Result[list["Change"], ErrorsList]: from donna.machine.changes import ChangeAddWorkUnit, ChangeSetTaskContext meta = cast(RunScriptMeta, operation.meta) @@ -157,22 +161,43 @@ def execute_section(self, task: "Task", unit: "WorkUnit", operation: ArtifactSec script = meta.script assert script is not None + machine_journal.add( + actor_id="donna", + message=f"Run script `{operation.title}`", + current_task_id=str(task.id), + current_work_unit_id=str(unit.id), + current_operation_id=unit.operation_id, + ).unwrap() + stdout, stderr, exit_code = _run_script( script=script, timeout=meta.timeout, project_dir=workspace_config.project_dir(), ) + machine_journal.add( + actor_id="donna", + message=( + f"Script finished `{operation.title}`, exit code: {exit_code}, " + f"has stdout: {bool(stdout)}, has stderr: {bool(stderr)}`" + ), + current_task_id=str(task.id), + current_work_unit_id=str(unit.id), + current_operation_id=unit.operation_id, + ).unwrap() + + changes: list["Change"] = [] if meta.save_stdout_to is not None: - yield ChangeSetTaskContext(task_id=task.id, key=meta.save_stdout_to, value=stdout) + changes.append(ChangeSetTaskContext(task_id=task.id, key=meta.save_stdout_to, value=stdout)) if meta.save_stderr_to is not None: - yield ChangeSetTaskContext(task_id=task.id, key=meta.save_stderr_to, value=stderr) + changes.append(ChangeSetTaskContext(task_id=task.id, key=meta.save_stderr_to, value=stderr)) next_operation = meta.select_next_operation(exit_code) full_operation_id = unit.operation_id.full_artifact_id.to_full_local(next_operation) - yield ChangeAddWorkUnit(task_id=task.id, operation_id=full_operation_id) + changes.append(ChangeAddWorkUnit(task_id=task.id, operation_id=full_operation_id)) + return Ok(changes) def validate_section( # noqa: CCR001 self, artifact: Artifact, section_id: ArtifactSectionId diff --git a/donna/protocol/cell_shortcuts.py b/donna/protocol/cell_shortcuts.py index 7dc75f4f..13ead514 100644 --- a/donna/protocol/cell_shortcuts.py +++ b/donna/protocol/cell_shortcuts.py @@ -7,3 +7,7 @@ def operation_succeeded(message: str, **meta: MetaValue) -> Cell: def operation_failed(message: str, **meta: MetaValue) -> Cell: return Cell.build(kind="operation_failed", media_type="text/markdown", content=message, **meta) + + +def info(message: str, **meta: MetaValue) -> Cell: + return Cell.build(kind="info", media_type="text/markdown", content=message, **meta) diff --git a/donna/protocol/formatters/automation.py b/donna/protocol/formatters/automation.py index 69854606..8f6bb6ff 100644 --- a/donna/protocol/formatters/automation.py +++ b/donna/protocol/formatters/automation.py @@ -1,12 +1,13 @@ import json +from donna.machine.journal import JournalRecord, serialize_record from donna.protocol.cells import Cell from donna.protocol.formatters.base import Formatter as BaseFormatter class Formatter(BaseFormatter): - def format_cell(self, cell: Cell, single_mode: bool) -> bytes: + def format_cell(self, cell: Cell) -> bytes: data: dict[str, str | int | bool | None] = {"id": cell.short_id} for meta_key, meta_value in sorted(cell.meta.items()): @@ -16,10 +17,5 @@ def format_cell(self, cell: Cell, single_mode: bool) -> bytes: return json.dumps(data, ensure_ascii=False, indent=None, separators=(",", ":"), sort_keys=True).encode() - def format_log(self, cell: Cell, single_mode: bool) -> bytes: - return self.format_cells([cell]) - - def format_cells(self, cells: list[Cell]) -> bytes: - single_mode = len(cells) == 1 - formatted_cells = [self.format_cell(cell, single_mode=single_mode) for cell in cells] - return b"\n".join(formatted_cells) + def format_journal(self, record: JournalRecord) -> bytes: + return serialize_record(record) diff --git a/donna/protocol/formatters/base.py b/donna/protocol/formatters/base.py index 938e50ac..0ffb1241 100644 --- a/donna/protocol/formatters/base.py +++ b/donna/protocol/formatters/base.py @@ -1,15 +1,13 @@ from abc import ABC, abstractmethod +from donna.machine.journal import JournalRecord from donna.protocol.cells import Cell class Formatter(ABC): @abstractmethod - def format_cell(self, cell: Cell, single_mode: bool) -> bytes: ... # noqa: E704 + def format_cell(self, cell: Cell) -> bytes: ... # noqa: E704 @abstractmethod - def format_log(self, cell: Cell, single_mode: bool) -> bytes: ... # noqa: E704 - - @abstractmethod - def format_cells(self, cells: list[Cell]) -> bytes: ... # noqa: E704 + def format_journal(self, record: JournalRecord) -> bytes: ... # noqa: E704 diff --git a/donna/protocol/formatters/human.py b/donna/protocol/formatters/human.py index 777842e8..a24fce5c 100644 --- a/donna/protocol/formatters/human.py +++ b/donna/protocol/formatters/human.py @@ -1,16 +1,14 @@ +from donna.machine.journal import JournalRecord from donna.protocol.cells import Cell from donna.protocol.formatters.base import Formatter as BaseFormatter class Formatter(BaseFormatter): - def format_cell(self, cell: Cell, single_mode: bool) -> bytes: + def format_cell(self, cell: Cell) -> bytes: id = cell.short_id - lines = [] - - if not single_mode: - lines = [f"----- DONNA CELL {id} -----"] + lines = [f"----- DONNA CELL {id} -----"] lines.append(f"kind = {cell.kind}") @@ -26,11 +24,9 @@ def format_cell(self, cell: Cell, single_mode: bool) -> bytes: return "\n".join(lines).encode() - def format_log(self, cell: Cell, single_mode: bool) -> bytes: - message = cell.content.strip() if cell.content else "" - return f"DONNA LOG: {message}".strip().encode() - - def format_cells(self, cells: list[Cell]) -> bytes: - single_mode = len(cells) == 1 - formatted_cells = [self.format_cell(cell, single_mode=single_mode) for cell in cells] - return b"\n\n".join(formatted_cells) + def format_journal(self, record: JournalRecord) -> bytes: + timestamp = record.timestamp.time().isoformat("seconds") + actor_id = record.actor_id or "-" + current_task_id = record.current_task_id.short if record.current_task_id is not None else "-" + output = f"{timestamp} [{current_task_id}] <{actor_id}> {record.message}" + return output.encode() diff --git a/donna/protocol/formatters/llm.py b/donna/protocol/formatters/llm.py index 1eee6748..c7a0c44c 100644 --- a/donna/protocol/formatters/llm.py +++ b/donna/protocol/formatters/llm.py @@ -1,16 +1,14 @@ +from donna.machine.journal import JournalRecord from donna.protocol.cells import Cell from donna.protocol.formatters.base import Formatter as BaseFormatter class Formatter(BaseFormatter): - def format_cell(self, cell: Cell, single_mode: bool) -> bytes: # noqa: CCR001 + def format_cell(self, cell: Cell) -> bytes: # noqa: CCR001 id = cell.short_id - lines = [] - - if not single_mode: - lines = [f"--DONNA-CELL {id} BEGIN--"] + lines = [f"--DONNA-CELL {id} BEGIN--"] lines.append(f"kind={cell.kind}") @@ -24,16 +22,23 @@ def format_cell(self, cell: Cell, single_mode: bool) -> bytes: # noqa: CCR001 lines.append("") lines.append(cell.content.strip()) - if not single_mode: - lines.append(f"--DONNA-CELL {id} END--") + lines.append(f"--DONNA-CELL {id} END--") return "\n".join(lines).strip().encode() - def format_log(self, cell: Cell, single_mode: bool) -> bytes: - message = cell.content.strip() if cell.content else "" - return f"DONNA LOG: {message}".strip().encode() - - def format_cells(self, cells: list[Cell]) -> bytes: - single_mode = len(cells) == 1 - formatted_cells = [self.format_cell(cell, single_mode=single_mode) for cell in cells] - return b"\n\n".join(formatted_cells) + def format_journal(self, record: JournalRecord) -> bytes: + timestamp = record.timestamp.isoformat() + actor_id = record.actor_id or "-" + current_task_id = record.current_task_id or "-" + current_work_unit_id = record.current_work_unit_id or "-" + current_operation_id = record.current_operation_id or "-" + + output = ( + f"{timestamp} " + f"[{current_task_id}] " + f"<{actor_id}> " + f"[{current_work_unit_id}] " + f"[{current_operation_id}] " + f"{record.message}" + ) + return output.encode() diff --git a/donna/protocol/utils.py b/donna/protocol/utils.py index 9ac4c5bc..648f3da0 100644 --- a/donna/protocol/utils.py +++ b/donna/protocol/utils.py @@ -1,22 +1,22 @@ import sys +from donna.machine.journal import JournalRecord from donna.protocol.cells import Cell from donna.protocol.modes import get_cell_formatter -def instant_output(cells: list[Cell]) -> None: - if not cells: - return +def instant_output(text: bytes) -> None: + sys.stdout.buffer.write(text + b"\n") + sys.stdout.buffer.flush() + +def instant_output_journal(record: JournalRecord) -> None: formatter = get_cell_formatter() + formatted_output = formatter.format_journal(record) + instant_output(formatted_output) - formatted_cells: list[bytes] = [] - for cell in cells: - # TODO: we should refactor that hardcoded check somehow - if cell.kind == "donna_log": - formatted_cells.append(formatter.format_log(cell, single_mode=True)) - else: - formatted_cells.append(formatter.format_cell(cell, single_mode=False)) - sys.stdout.buffer.write(b"\n\n".join(formatted_cells) + b"\n\n") - sys.stdout.buffer.flush() +def instant_output_cell(cell: Cell) -> None: + formatter = get_cell_formatter() + formatted_output = formatter.format_cell(cell) + instant_output(formatted_output) diff --git a/donna/workspaces/utils.py b/donna/workspaces/utils.py new file mode 100644 index 00000000..5ff41285 --- /dev/null +++ b/donna/workspaces/utils.py @@ -0,0 +1,15 @@ +from donna.core.errors import ErrorsList +from donna.core.result import Ok, Result, unwrap_to_error +from donna.domain.ids import WorldId +from donna.workspaces.config import config +from donna.workspaces.worlds.base import World + + +@unwrap_to_error +def session_world() -> Result[World, ErrorsList]: + world = config().get_world(WorldId("session")).unwrap() + + if not world.is_initialized(): + world.initialize(reset=False) + + return Ok(world) diff --git a/donna/workspaces/worlds/base.py b/donna/workspaces/worlds/base.py index 2d5fd0f9..6d1949fa 100644 --- a/donna/workspaces/worlds/base.py +++ b/donna/workspaces/worlds/base.py @@ -1,6 +1,7 @@ from __future__ import annotations from abc import ABC, abstractmethod +from collections.abc import Iterable from typing import TYPE_CHECKING from donna.core.entities import BaseEntity @@ -59,6 +60,16 @@ def read_state(self, name: str) -> Result[bytes | None, ErrorsList]: ... # noqa @abstractmethod def write_state(self, name: str, content: bytes) -> Result[None, ErrorsList]: ... # noqa: E704 + @abstractmethod + def journal_reset(self) -> Result[None, ErrorsList]: ... # noqa: E704 + + @abstractmethod + def journal_add(self, content: bytes) -> Result[None, ErrorsList]: ... # noqa: E704 + + @abstractmethod + def journal_read(self, lines: int | None = None, follow: bool = False) -> Iterable[Result[bytes, ErrorsList]]: + pass + def initialize(self, reset: bool = False) -> None: pass diff --git a/donna/workspaces/worlds/filesystem.py b/donna/workspaces/worlds/filesystem.py index a537ee5a..827f5a22 100644 --- a/donna/workspaces/worlds/filesystem.py +++ b/donna/workspaces/worlds/filesystem.py @@ -1,5 +1,9 @@ +import os import pathlib import shutil +import stat +import time +from collections.abc import Iterable from typing import TYPE_CHECKING, cast from donna.core.errors import ErrorsList @@ -18,6 +22,10 @@ class World(BaseWorld): path: pathlib.Path + _journal_file_name = "journal.jsonl" + + def _journal_path(self) -> pathlib.Path: + return self.path / self._journal_file_name def _artifact_listing_root(self) -> ArtifactListingNode | None: if not self.path.exists(): @@ -168,6 +176,118 @@ def write_state(self, name: str, content: bytes) -> Result[None, ErrorsList]: path.write_bytes(content) return Ok(None) + def journal_reset(self) -> Result[None, ErrorsList]: + if self.readonly: + return Err([world_errors.WorldReadonly(world_id=self.id)]) + + if not self.session: + return Err([world_errors.WorldStateStorageUnsupported(world_id=self.id)]) + + path = self._journal_path() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(b"") + return Ok(None) + + def journal_add(self, content: bytes) -> Result[None, ErrorsList]: + if self.readonly: + return Err([world_errors.WorldReadonly(world_id=self.id)]) + + if not self.session: + return Err([world_errors.WorldStateStorageUnsupported(world_id=self.id)]) + + path = self._journal_path() + path.parent.mkdir(parents=True, exist_ok=True) + + with path.open("ab") as stream: + stream.write(content.rstrip(b"\n")) + stream.write(b"\n") + + return Ok(None) + + def _journal_read_all(self, path: pathlib.Path) -> list[bytes]: + if not path.exists(): + return [] + + with path.open("rb") as stream: + return [line.rstrip(b"\n") for line in stream if line.strip()] + + def _journal_file_identity(self, path: pathlib.Path) -> tuple[int, int] | None: + try: + path_stat = path.stat() + except FileNotFoundError: + return None + + if not stat.S_ISREG(path_stat.st_mode): + return None + + return (path_stat.st_dev, path_stat.st_ino) + + def _journal_follow( # noqa: CCR001 + self, + poll_interval: float = 0.25, + ) -> Iterable[Result[bytes, ErrorsList]]: + path = self._journal_path() + + stream = None + stream_identity: tuple[int, int] | None = None + + # if the journal file did exist when we started following, we want to read from the end + # if the journal file didn't exist when we started following, we want to read from the start + start_from_head = False + + while True: + file_identity = self._journal_file_identity(path) + + if stream is not None and stream_identity != file_identity: + stream.close() + stream = None + stream_identity = None + + if file_identity is None or file_identity == stream_identity: + start_from_head = True + + if stream is None and file_identity is not None: + stream = path.open("rb") + + if not start_from_head: + stream.seek(0, os.SEEK_END) + + stream_identity = file_identity + + if stream is None: + time.sleep(poll_interval) + continue + + while line := stream.readline(): + line = line.rstrip(b"\n") + if line.strip(): + yield Ok(line) + + time.sleep(poll_interval) + + def _journal_read_some(self, lines: int | None = None) -> Iterable[Result[bytes, ErrorsList]]: + path = self._journal_path() + + records = self._journal_read_all(path) + + if lines is not None: + records = records[-lines:] if lines > 0 else [] + + for record in records: + yield Ok(record) + + def journal_read(self, lines: int | None = None, follow: bool = False) -> Iterable[Result[bytes, ErrorsList]]: + if not self.session: + yield Err([world_errors.WorldStateStorageUnsupported(world_id=self.id)]) + return + + yield from self._journal_read_some(lines=lines) + + if not follow: + return + + yield from self._journal_follow() + def list_artifacts(self, pattern: FullArtifactIdPattern) -> list[ArtifactId]: # noqa: CCR001 return list_artifacts_by_pattern( world_id=self.id, diff --git a/donna/workspaces/worlds/python.py b/donna/workspaces/worlds/python.py index 5bad92fc..2c67153e 100644 --- a/donna/workspaces/worlds/python.py +++ b/donna/workspaces/worlds/python.py @@ -1,6 +1,7 @@ import importlib import importlib.resources import pathlib +from collections.abc import Iterable from typing import TYPE_CHECKING, cast from donna.core.errors import ErrorsList @@ -167,6 +168,15 @@ def read_state(self, name: str) -> Result[bytes | None, ErrorsList]: def write_state(self, name: str, content: bytes) -> Result[None, ErrorsList]: return Err([world_errors.WorldStateStorageUnsupported(world_id=self.id)]) + def journal_reset(self) -> Result[None, ErrorsList]: + return Err([world_errors.WorldStateStorageUnsupported(world_id=self.id)]) + + def journal_add(self, content: bytes) -> Result[None, ErrorsList]: + return Err([world_errors.WorldStateStorageUnsupported(world_id=self.id)]) + + def journal_read(self, lines: int | None = None, follow: bool = False) -> Iterable[Result[bytes, ErrorsList]]: + yield Err([world_errors.WorldStateStorageUnsupported(world_id=self.id)]) + def initialize(self, reset: bool = False) -> None: pass