You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sanghyuk Kim (Mass General Brigham & University of Massachusetts, USA)
Project Description
The Slicer AI Agent is a scripted extension that lets users use natural-language instructions and have the system autonomously generate, validate, and execute Python code within the 3D Slicer scene. For complex multi-step operations, such as extension-specific surgical planning workflows, the agent enters a deterministic workflow runtime that executes a sequence of predefined steps (automated code execution, interactive 3D mark-up placement, user choices, and branching logic) with a progress bar tracking completion.
Currently this progress bar is forward-only: once a step completes, there is no way to go back, inspect what happened at an earlier step, or modify a previous choice and re-run the downstream pipeline. This limits both transparency (users cannot easily understand what was done and why) and usability (a wrong choice means restarting the entire workflow from scratch).
We propose adding a workflow replay timeline that records per-step state and allows users to go back to any completed step, inspect the code and choices that were made, optionally modify parameters, and re-execute the workflow from that point onward.
Objective
We aim to add a workflow replay and step-back navigation capability to the Slicer AI Agent's workflow runtime. By recording per-step state and exposing it through an interactive timeline, users will be able to scrub back to any completed step, inspect what happened, optionally modify a prior choice, and re-run the remaining pipeline — improving both process transparency and the ability to correct mistakes without restarting the workflow from scratch.
Approach and Plan
Step-level recording: Capture a full replay log for every workflow step, including the generated code, user choices, scene snapshots (before/after), and execution results, so that the complete history of a workflow run is persisted and inspectable.
Timeline UI with scrubbing: Replace the current linear progress bar with an interactive timeline widget that shows step markers (completed / current / pending), allows dragging to any completed step, and displays a detail panel with the code, choices, and scene diff for that step.
Step-back re-execution: Enable the user to jump back to a completed step, optionally modify a prior choice or parameter, and re-run the remaining workflow from that point, reusing the existing template-dispatch and auto-advance infrastructure.
Draft Status
Draft - team will hold off on page creation
Category
Infrastructure
Key Investigators
Project Description
The Slicer AI Agent is a scripted extension that lets users use natural-language instructions and have the system autonomously generate, validate, and execute Python code within the 3D Slicer scene. For complex multi-step operations, such as extension-specific surgical planning workflows, the agent enters a deterministic workflow runtime that executes a sequence of predefined steps (automated code execution, interactive 3D mark-up placement, user choices, and branching logic) with a progress bar tracking completion.
Currently this progress bar is forward-only: once a step completes, there is no way to go back, inspect what happened at an earlier step, or modify a previous choice and re-run the downstream pipeline. This limits both transparency (users cannot easily understand what was done and why) and usability (a wrong choice means restarting the entire workflow from scratch).
We propose adding a workflow replay timeline that records per-step state and allows users to go back to any completed step, inspect the code and choices that were made, optionally modify parameters, and re-execute the workflow from that point onward.
Objective
We aim to add a workflow replay and step-back navigation capability to the Slicer AI Agent's workflow runtime. By recording per-step state and exposing it through an interactive timeline, users will be able to scrub back to any completed step, inspect what happened, optionally modify a prior choice, and re-run the remaining pipeline — improving both process transparency and the ability to correct mistakes without restarting the workflow from scratch.
Approach and Plan
Step-level recording: Capture a full replay log for every workflow step, including the generated code, user choices, scene snapshots (before/after), and execution results, so that the complete history of a workflow run is persisted and inspectable.
Timeline UI with scrubbing: Replace the current linear progress bar with an interactive timeline widget that shows step markers (completed / current / pending), allows dragging to any completed step, and displays a detail panel with the code, choices, and scene diff for that step.
Step-back re-execution: Enable the user to jump back to a completed step, optionally modify a prior choice or parameter, and re-run the remaining workflow from that point, reusing the existing template-dispatch and auto-advance infrastructure.
Progress and Next Steps
TBD
Illustrations
bone_reconstruction_compressed.mp4
pelvic_compressed.mp4
Background and References
Slicer AI Agent source repository: https://github.com/puxuntu/Slicer_agent