planner

[esokullu]

if we had a planner, it would know to schedule tasks, it would know what tools to use a priori and confirm the plan with the user.
like:
devise a plan based on this prompt:
if there is a time related .. do schedule it like
...
use scratchpad to ..
use progress_.. to if the task involves note taking because they won't get lost with compaction.

[Ayush7614]

Assessment: yes — substantial but much of the plumbing already exists

Hello @esokullu I like to work on this issue

The issue describes a pre-execution planning phase that (a) picks the right infrastructure tools (schedule_task, scratchpad_write, progress_update, …), (b) proposes a step plan, and (c) gets user confirmation before Act mode runs.

That does not exist as a dedicated subsystem today. What we do have is reactive guidance in the Act system prompt + individual tools the model may discover mid-run. A planner would make those choices explicit and user-visible up front.

---

What already exists (building blocks)

Capability	Current state
Scheduling	schedule_task (one-shot/recurring jobs) + schedule_resume (pause current run) + full scheduler.js queue/alarm runner
Durable notes	scratchpad_write — pinned message surviving compaction
Structured progress	progress_update / progress_read + progress ledger pinned in context; blocks done if rows unresolved
User confirmation	clarify (pause run, wait for answer) — pattern for human-in-the-loop
Lightweight planning UX	/plan slash → Ask mode + prefix "Plan the following step by step:" (no structured output, no confirm gate)
Vision sidecar	Dedicated vision model described as feeding the "planning agent"
Site adapters	URL-specific hints injected at run start

So the issue is less "build scheduling from scratch" and more orchestrate + surface decisions the prompts already try to teach reactively.

---

Proposed architecture

Phase 1 — Planner pass + confirm gate (MVP)

Trigger (opt-in setting): Plan before Act in Settings, or per-message via /plan reworked to mean "plan then execute".

Flow:

User message (Act mode)
  → Planner pass (read-only tools only OR no tools — just prompt + page context)
  → Structured plan JSON
  → Side panel plan card: [Approve] [Edit] [Cancel]
  → On Approve:
      scratchpad_write({ text: formatted plan, replace: true })
      start normal Act agent loop (plan now pinned in context)

Planner output schema (example):

{
  "summary": "Follow GitHub stargazers who match …",
  "steps": [
    { "id": "1", "action": "Open stargazers list", "tools": ["navigate", "wait_for_stable", "get_accessibility_tree"] },
    { "id": "2", "action": "For each user: click Follow, record status", "tools": ["click_ax", "progress_update"] }
  ],
  "memory": {
    "scratchpad": ["running counts", "download IDs", "file paths"],
    "progress_ledger": true,
    "progress_action": "follow"
  },
  "scheduling": null,
  "mode": "act",
  "risks": ["bulk follow actions — confirm scope"],
  "estimated_tool_calls": "high"
}

Scheduling heuristics in planner prompt (not hard-coded logic in v1):

• User says "tomorrow", "every hour", "remind me", "monitor" → recommend schedule_task with target.type:"url"
• Current run blocked on CI/deploy/email → recommend schedule_resume (not schedule_task)
• Seconds-level SPA wait → wait_for_stable / wait_for_element (never schedule)

Confirmation UX: Reuse the clarify pause pattern or a new plan_review UI card (cleaner — doesn't consume a clarify budget). Approve = user message "[Plan approved]" or explicit button → Act loop.

Phase 2 — Tool-aware planner

• Planner prompt includes compressed tool catalog (names + one-line when-to-use), not full JSON schemas — keeps token cost low.
• Flag when task needs tools the current provider can't support (e.g. upload_file on Firefox, execute_js on Chrome).
• Auto-suggest Compact mode for small local models if plan > N steps.

Phase 3 — Dedicated planning model

• Settings: Planning model (like existing dedicated vision model) — cheap text model plans, active provider executes.
• Matches landing-page "fast model for planning + vision model for screenshots" story.

---

Why scratchpad + progress_ledger in the plan matters

The issue author is right: compaction is the failure mode for long tasks. Today the Act prompt tells the model to use scratchpad/progress tools, but models still forget until step 15+.

Planner should commit upfront:

• scratchpad for misc durable facts (paths, IDs, "step 2 of 5 done")
• progress_ledger for repeated item/action loops (one row per user/email/result) — this is app-owned state with done-blocking semantics, stronger than scratchpad for lists

Planner output should literally say: "This task uses progress_ledger with action=follow; do not track items only in scratchpad."

---

Implementation touchpoints

Layer	Files
Planner module	src/chrome/src/agent/planner.js (new) — buildPlan(tabId, userMessage, pageContext)
Agent integration	agent.js — branch in processMessage before tool loop when planning enabled
Schema / prompt	tools.js or planner-prompt.js — PLANNER_SYSTEM_PROMPT + JSON schema
UI	sidepanel.js — plan review card + approve/edit buttons
Settings	settings.js + settings.html — toggles for plan-before-act, planning model
i18n	locales/*.js
Firefox	mirror planner module + UI

No new manifest permissions. Planner pass is read-only until user approves.

---

Scope / risk

MVP estimate	~400–800 lines (Chrome), +Firefox mirror
Risk	Extra latency + token cost per Act run; mitigate with opt-in + cheap planning model
Security	Plan is model-generated — show risks section; destructive steps need explicit approve (already Act-mode gated)
Testing	Fixture scenarios: "follow 10 users" → plan mentions progress_ledger; "check tomorrow" → plan mentions schedule_task

---

Relation to /plan today

Current /plan only rewrites the user message in Ask mode. Recommendation:

• Keep Ask-only /plan for "just give me a plan, don't act"
• Add Plan + Execute path: /plan with opt-in auto-act after approve, or Settings toggle for all Act runs

---

Suggested milestone split

1. v1: structured plan + UI confirm + auto scratchpad_write on approve
2. v1.1: scheduling + memory-strategy fields in plan schema
3. v2: dedicated planning model + progress_ledger auto-seeding from plan
4. v3: edit plan mid-run ("replan" tool when stuck)

Happy to prototype v1 if we want a concrete PR to discuss against.

[esokullu]

greeat points, feel free to implement in a new branch and we'll merge it