include shortcodes + source-info infrastructure

.beads/issues.jsonl

@@ -19,6 +19,7 @@
 {"id":"bd-2gc9","title":"Nested tight lists incorrectly marked as loose (Para instead of Plain)","description":"In process_list() at crates/pampa/src/pandoc/treesitter.rs:183-192, multi-block list items with at least one paragraph are unconditionally marked as loose. This is wrong: a list item containing [Paragraph, BulletList] (e.g., '* foo\\n  * bar') has no blank line between blocks and should remain tight. The CommonMark spec says a list is loose only when items are separated by blank lines or contain two block elements WITH A BLANK LINE between them.","status":"closed","priority":1,"issue_type":"bug","created_at":"2026-02-13T22:49:14.652871Z","created_by":"cscheid","updated_at":"2026-02-13T23:05:58.650439Z","closed_at":"2026-02-13T23:05:58.650417Z","close_reason":"Fixed: detect blank lines via tree-sitter block_continuation spans in process_list_item, propagate through IntermediateListItem","source_repo":".","compaction_level":0,"original_size":0}
 {"id":"bd-2gkx","title":"Kanban: New card creation button with form","description":"Add a 'New Card' button that opens a form with: title (required), type dropdown (feature/milestone/bug/task), optional deadline with date picker, optional status. Default creation date to today. Extend addCard() in astHelpers.ts to support deadline and status. Plan: claude-notes/plans/2026-02-11-kanban-ui-enhancements.md","status":"closed","priority":2,"issue_type":"feature","created_at":"2026-02-11T18:32:46.896753Z","created_by":"cscheid","updated_at":"2026-02-11T18:39:33.119846Z","closed_at":"2026-02-11T18:39:33.119829Z","close_reason":"Implemented - new card form with type, status, and deadline","source_repo":".","compaction_level":0,"original_size":0}
 {"id":"bd-2h6x","title":"Slide thumbnails show in outline pane for non-slide documents","description":"The useSlideThumbnails hook runs unconditionally for all documents, generating thumbnails even for regular HTML documents. PreviewRouter detects format: q2-slides but doesn't communicate the format back to Editor.tsx. Fix: add onFormatChange callback from PreviewRouter so Editor can conditionally generate thumbnails. Plan: claude-notes/plans/2026-02-26-slide-thumbnails-conditional.md","status":"closed","priority":2,"issue_type":"bug","created_at":"2026-02-26T15:04:39.650792Z","created_by":"cscheid","updated_at":"2026-02-26T15:10:26.512270Z","closed_at":"2026-02-26T15:10:26.512246Z","close_reason":"Fixed: slide thumbnails now conditional on format: q2-slides","source_repo":".","compaction_level":0,"original_size":0}
+{"id":"bd-2mxo","title":"Metadata materialization drops source_info provenance","description":"MergedConfig::materialize() in quarto-config/src/materialize.rs loses source provenance that pampa's YAML parser correctly provides:\n- Map container source_info replaced with heuristic/default (lines ~121-161)\n- Map entry key_source always set to SourceInfo::default() (line ~132)\n- Array container source_info uses only last item (lines ~109-113)\n\nScalar values are preserved correctly. Affects error reporting for any metadata-related diagnostics after the merge stage. The code has an explicit comment acknowledging the key_source loss: 'We lose key source info during materialization.'\n\nPre-existing since commit 955bc326 (2025-12-07, 'config merging in'). Not blocking Plan 0 but degrades the YAML frontmatter Concat piece in the QMD writer's SourceInfo output.","status":"open","priority":2,"issue_type":"bug","created_at":"2026-04-20T15:52:16.644685Z","created_by":"gordon","updated_at":"2026-04-20T15:52:16.644685Z","source_repo":".","compaction_level":0,"original_size":0}
 {"id":"bd-2olu","title":"Hub MCP Server: automerge project access for AI agents","description":"Design and implement an MCP server that allows AI coding agents (Claude Code, Codex, etc.) to read and write files in Quarto Hub projects via automerge sync, without requiring filesystem access. Plan: claude-notes/plans/2026-03-13-hub-mcp-server-design.md","status":"open","priority":1,"issue_type":"feature","created_at":"2026-03-13T23:38:33.161278Z","created_by":"cscheid","updated_at":"2026-03-13T23:38:33.161278Z","source_repo":".","compaction_level":0,"original_size":0}
 {"id":"bd-2rbk","title":"Improve pampa test skip behavior when Pandoc is absent","description":"4 pampa tests hard-assert instead of gracefully skipping when Pandoc is not available. Should use test skip or conditional compilation so cargo xtask test runs cleanly without Pandoc. Low priority since pampa is currently excluded from Windows test compilation due to v8.","status":"open","priority":3,"issue_type":"task","created_at":"2026-03-20T13:36:13.451656600Z","created_by":"cderv","updated_at":"2026-03-20T13:36:13.451656600Z","source_repo":".","compaction_level":0,"original_size":0}
 {"id":"bd-2s6j","title":"Kanban: Horizontal rows instead of columns","description":"Change the board layout from vertical status columns to horizontal rows. Each status group becomes a full-width row with cards flowing left-to-right. Plan: claude-notes/plans/2026-02-11-kanban-ui-enhancements.md","status":"closed","priority":2,"issue_type":"task","created_at":"2026-02-11T18:32:33.985420Z","created_by":"cscheid","updated_at":"2026-02-11T18:34:29.136267Z","closed_at":"2026-02-11T18:34:29.136249Z","close_reason":"Implemented - board now uses horizontal rows","source_repo":".","compaction_level":0,"original_size":0}

CLAUDE.md

@@ -30,7 +30,7 @@ Always follow TDD workflow: write/update tests BEFORE implementing features. Whe
 1. Stage and commit changes as needed
 2. **Verify the full workspace compiles cleanly** (`cargo build --workspace`)
 3. **Verify the full workspace tests pass** (`cargo nextest run --workspace`)
-4. **For changes to quarto-core or quarto-pandoc-types**: Run `cargo xtask verify` to ensure hub-client/WASM builds work
+4. **Run `cargo xtask verify`** — at minimum `cargo xtask verify --skip-hub-build` for Rust-only changes; full `cargo xtask verify` when the WASM leg could be affected (any change under `quarto-core`, `quarto-pandoc-types`, or anything else hub-client depends on). This is the step that matches CI's `-D warnings` strictness; plain `cargo build` / `cargo nextest` from steps 2 and 3 do not.
 5. Ask the user for permission before pushing
 6. Only push after receiving explicit approval
 

claude-notes/plans/2026-03-16-extensions-grand-plan.md

@@ -447,14 +447,18 @@ shortcode processing pipeline. Includes block-level shortcode support and a new
 
 **Goal**: Extensions can provide custom execution engines.
 
-- [ ] Parse `contributes.engines` from extensions
-- [ ] Register engines in the engine execution stage
-- [ ] Support external engine paths
-- [ ] Tests
-
-**Open questions**:
-- What engines does q2 currently support?
-- How is the engine selection mechanism implemented?
+**Status**: Superseded by the TypeScript Engine Extensions grand plan:
+`claude-notes/plans/2026-04-16-ts-engine-extensions-subprocess.md`
+
+That plan covers engine extension parsing (adding `engines` to `Contributes`),
+subprocess-based execution via Deno, engine discovery/claiming, and registration.
+Plan 1 Phase 1D within it handles the `_extension.yml` parsing and `EngineRegistry`
+integration that was originally scoped here.
+
+**Answered open questions**:
+- q2 currently supports markdown, knitr, and jupyter engines (all built-in)
+- Engine selection uses metadata-based detection (`detect_engine()` in `detection.rs`);
+  the TS engine plan adds a 4-phase claiming algorithm (file ext → YAML → language scan → fallback)
 
 ### Phase 9: Embedded Extensions
 

claude-notes/plans/2026-04-16-julia-validation.md

@@ -0,0 +1,203 @@
+# Plan 4: Julia Engine Validation
+
+**Grand plan:** [2026-04-16-ts-engine-extensions-subprocess.md](2026-04-16-ts-engine-extensions-subprocess.md)
+**Depends on:** Plans 1, 2, and 3 (all must be substantially complete)
+**Blocks:** Nothing (this is the final validation plan)
+**Estimated sessions:** 1-2
+
+## Overview
+
+End-to-end validation of the TypeScript engine extension system using the Julia engine from Quarto 1. Take the real `julia-engine.ts`, set it up as a q2 extension, and render documents with Julia code cells.
+
+This plan is primarily integration debugging. If Plans 1a, 1b, 1c, 2, and 3 are solid and the echo engine test from Plan 1c Phase 3 passes, most of the infrastructure works. This plan surfaces the gaps specific to a real-world engine extension.
+
+## Prerequisites
+
+- [ ] Plans 1a, 1b, and 1c complete: Rust subprocess infrastructure + Deno harness + extension integration, echo engine passes
+- [ ] Plan 2 complete: `@quarto/api` package with text/markdown/format/path/system/console/crypto subpaths, all QuartoAPI namespaces except `jupyter` wired in
+- [ ] Plan 3 complete: `@quarto/api/jupyter` with `toMarkdown` working and wired into engine-host
+- [ ] Julia installed on the test machine (`julia` in PATH)
+
+## Work Items
+
+### Phase 4A: Set up Julia engine extension
+
+- [ ] Copy Julia engine from Quarto 1's **source/development version** (NOT the pkg-working version):
+  ```
+  ~/src/quarto-cli/src/resources/extension-subtrees/julia-engine/
+  ```
+  Use this version because it resolves resource files via `import.meta.url` (relative to the JS file), rather than the distributed version which uses `quarto.path.resource()` pointing to Quarto's global `share/` directory.
+
+- [ ] Create test fixture with engine source AND its resource files:
+  ```
+  tests/fixtures/extensions/julia-engine/
+    _extension.yml
+    src/
+      julia-engine.ts
+      constants.ts
+    Project.toml                        ← Julia environment definition
+    ensure_environment.jl               ← Julia setup script
+    quartonotebookrunner.jl             ← Julia execution entry point
+    start_quartonotebookrunner_detached.jl  ← Daemon launcher
+  ```
+  The .jl files and Project.toml live alongside the extension (same directory or parent) so that `dirname(import.meta.url)` resolves them. This matches Quarto 1's development-mode layout where the extension is self-contained.
+- [ ] Write `_extension.yml`:
+  ```yaml
+  title: Julia Engine
+  author: Quarto
+  version: 1.0.0
+  contributes:
+    engines:
+      - name: julia
+        path: src/julia-engine.ts
+  ```
+- [ ] Identify needed modifications to `julia-engine.ts`:
+  - Import paths: change `@quarto/types` imports if our type names differ
+  - API calls: verify all `quarto.*` calls match our implementation signatures
+  - Resource resolution: verify `import.meta.url`-based paths work after bundling (the bundled .js file's URL determines the base directory — resource files must be relative to where the bundle is loaded from)
+  - Deno APIs: verify `Deno.Command`, `Deno.connect`, `crypto.subtle`, file I/O all work (they should — it's running in real Deno)
+  - Standard library imports: `"path"`, `"fs/exists"`, `"encoding/base64"` — resolved at build time via the import map
+- [ ] Document every modification in a compatibility log
+
+### Phase 4B: Minimal Julia render
+
+The simplest possible Julia document.
+
+- [ ] Create test document:
+  ```markdown
+  ---
+  engine: julia
+  ---
+
+  ```{julia}
+  1 + 1
+  ```
+  ```
+- [ ] Run through q2's render pipeline. Use `cargo run -- render <file.qmd>` (the `quarto` crate at `crates/quarto/` is the main CLI binary). Check existing smoke tests in `crates/quarto/tests/` for how integration tests invoke rendering programmatically.
+- [ ] Debug the first failure. Common failure checklist:
+  - [ ] Extension not discovered → `_extension.yml` parsing issue
+  - [ ] Deno subprocess won't start → Deno not in PATH, or engine-host-deno bundle issue
+  - [ ] Engine module fails to load → import resolution, transpilation issue
+  - [ ] `engine.init()` fails → QuartoAPI construction issue
+  - [ ] `engine.launch()` fails → EngineProjectContext mismatch
+  - [ ] Julia process won't start → `Deno.Command` issue, Julia not in PATH
+  - [ ] Julia server connection fails → TCP connect issue, HMAC auth issue
+  - [ ] Execution succeeds but output is wrong → `toMarkdown()` issue
+  - [ ] Result deserialization fails → protocol/type mismatch
+- [ ] Iterate until the simple document renders successfully
+- [ ] Verify output HTML contains the result `2`
+
+### Phase 4C: Julia with figures
+
+- [ ] Create test document with a plot:
+  ```markdown
+  ---
+  engine: julia
+  ---
+
+  ```{julia}
+  using Plots
+  plot(1:10, rand(10))
+  ```
+  ```
+- [ ] Verify:
+  - [ ] Figure file generated in `_files/` directory
+  - [ ] Figure referenced correctly in output markdown
+  - [ ] `supporting` files tracked in `ExecuteResult`
+  - [ ] HTML output renders with the figure
+
+### Phase 4D: Multiple cells and error handling
+
+- [ ] Test multiple code cells:
+  ```markdown
+  ---
+  engine: julia
+  ---
+
+  ```{julia}
+  x = 42
+  ```
+
+  ```{julia}
+  println("x is $x")
+  ```
+  ```
+- [ ] Verify state persists between cells (x defined in first, used in second)
+
+- [ ] Test error handling:
+  ```markdown
+  ---
+  engine: julia
+  ---
+
+  ```{julia}
+  error("this should fail gracefully")
+  ```
+  ```
+- [ ] Verify error produces a useful message, not a crash
+
+### Phase 4E: Julia-specific features
+
+- [ ] Test daemon mode (`execute.daemon: true`) — Julia server stays alive
+- [ ] Test `exeflags` option — arguments passed to Julia
+- [ ] Test `env` option — environment variables set for Julia
+- [ ] Test cell options: `echo: false`, `output: false`, `warning: false`
+
+### Phase 4F: Regression audit
+
+- [ ] Run same test documents through Quarto 1 for comparison
+- [ ] Document output differences
+- [ ] Verify all existing q2 tests pass (`cargo nextest run --workspace`)
+- [ ] Run `cargo xtask verify` for full validation
+- [ ] File issues (via `br create`) for any gaps discovered
+
+### Phase 4G: Adaptation documentation
+
+- [ ] Write a summary of all changes needed to `julia-engine.ts`
+- [ ] Categorize changes:
+  - Import path adjustments
+  - API signature differences
+  - Missing QuartoAPI methods (if any were stubbed)
+  - Behavioral differences
+- [ ] This becomes the basis for documentation for extension authors migrating from Quarto 1
+
+## Design Notes
+
+### Debugging approach
+
+The subprocess architecture helps debugging — you can run the Deno engine-host independently:
+
+```bash
+# Run engine-host manually for debugging
+echo '{"type":"init","enginePath":"./julia-engine.ts","context":{...}}' | \
+  deno run --allow-all ts-packages/quarto-engine-host-deno/src/host.ts
+```
+
+You can also add `console.error()` statements in the engine or harness and see them on stderr.
+
+### Standard library imports
+
+The Julia engine imports `"path"`, `"fs/exists"`, `"encoding/base64"` from Deno's standard library. Following Quarto 1's approach, these are resolved at **build time** via the import map (`"path"` → `jsr:@std/path`, etc.) and inlined into the bundled `.js` file. At runtime, no import resolution is needed.
+
+The build step for the Julia engine fixture:
+```bash
+deno bundle --config=resources/extension-build/deno.json julia-engine.ts > julia-engine.js
+```
+
+### CI gating
+
+Julia engine tests should be:
+- Gated behind a feature flag or test tag (Julia may not be installed in CI)
+- Run manually during development
+- Optionally run in CI if Julia is available
+
+## Success Criteria
+
+- [ ] Julia engine extension discovered and loaded by q2
+- [ ] Simple Julia code cell executes and produces correct output
+- [ ] Figure generation works
+- [ ] Multiple cells with shared state work
+- [ ] Error handling produces useful messages
+- [ ] All modifications to julia-engine.ts documented
+- [ ] No regressions in existing tests
+- [ ] `cargo xtask verify` passes