Software Coach

中文 README | Deutsche README

Developer documentation. For the end-user guide shown inside the app, use:

• USER_GUIDE.md
• USER_GUIDE.zh-CN.md
• USER_GUIDE.de.md

Software Coach is a Windows-first, privacy-first, plugin-first desktop MVP for guided software coaching. It does not automate clicks. It resolves software context at the conversation level, sends screenshots directly to a vision-capable model when available, grounds the result in local workflow knowledge, and returns bounded next-step guidance with explicit evidence and uncertainty.

This repository currently ships one real Coach Pack: PerfView. The core is intentionally software-agnostic, model-provider-agnostic, and structured so that adding a new pack does not require changing core orchestration.

Project Philosophy

• Local-first: mock mode and local-provider mode remain useful offline.
• Privacy-first: screenshots are not uploaded by default.
• Plugin-first: software-specific logic lives in Coach Packs, not in core.
• Provider-agnostic: model integration is behind a single ModelProvider interface.
• Progressive coaching: the app is a step-by-step coach, not just a one-turn chatbot.

Architecture Rationale

This project uses a plugin-first monolith instead of a distributed plugin runtime.

• It keeps the MVP shippable.
• It preserves clean boundaries.
• It avoids leaking PerfView-specific logic into the core.
• It leaves room for future installable packs, OCR, trace analysis, and vector retrieval.

flowchart LR
  UI["React Overlay UI"] --> IPC["Typed Preload IPC"]
  IPC --> MAIN["Electron Main Process"]
  MAIN --> APP["Application Use Cases"]
  APP --> DOMAIN["Domain Contracts"]
  APP --> INFRA["Infrastructure Adapters"]
  APP --> PACKS["Coach Packs"]
  PACKS --> KNOW["Docs / Workflows / Ontology / Prompts / Fixtures"]
  INFRA --> MODELS["Model Providers"]
  INFRA --> STORE["Local Session / Feedback / Logs"]

Loading

Why plugin-first architecture

• New software integrations should be added by registering a new CoachPack.
• Core orchestration should not gain if (perfview) branches.
• Software-specific prompts, docs, workflows, ontology, fixtures, and future state graphs stay owned by the pack.

Why provider abstraction exists

• Different deployments need different model backends.
• Some users need offline or local-only operation.
• Some providers support vision, streaming, or embeddings differently.
• The app should be able to add Claude, Azure OpenAI, LM Studio, vLLM, or a local gateway without rewriting application logic.

Electron security decisions

Current packaged runtime:

• contextIsolation: true
• nodeIntegration: false
• typed preload API only
• CSP on the renderer page
• no dynamic remote script execution
• no arbitrary remote BrowserWindow navigation
• narrow IPC channels
• permission handler denies unused permissions

Current Windows packaging note:

• sandbox is temporarily disabled in the renderer for packaged builds because the MVP hit a real preload bridge failure in the packaged Electron runtime.
• This is a pragmatic compatibility tradeoff, not the final target architecture.
• The intended future state is to restore sandbox: true once the preload path is stable in packaged production builds.

More detail is in SECURITY.md.

Privacy guarantees

By default:

• screenshots stay local
• screenshots are not stored
• telemetry is disabled
• remote docs are disabled
• cloud model use is disabled

What leaves the machine:

• nothing in mock mode
• nothing in local Ollama mode, assuming Ollama runs locally
• text and image content may leave the machine only if the user enables a cloud-capable provider and selects it

What stays local:

• local docs, workflows, ontology, prompts, and fixtures
• feedback files
• sessions
• logs
• screenshots unless a future feature changes that behavior

Offline-first philosophy

The MVP is designed to remain useful offline:

• mock provider works without network or API keys
• local provider mode can run through an OpenAI-compatible local endpoint or Ollama
• local retrieval works from checked-in markdown and YAML assets

Conversation-first, vision-first flow

The current coaching flow is:

1. confirm software name + version in the conversation when possible
2. if the user does not provide them, ask for confirmation before detailed coaching
3. if the user still declines, use the first screenshot to produce a candidate software/version guess
4. pre-route a candidate Coach Pack
5. send the screenshot, question, and pack-grounded docs/workflows/prompts to a vision-capable model
6. validate the structured result and render the final coach response

This keeps the experience close to direct screenshot Q&A while preserving Coach Pack routing, local docs grounding, and progressive coaching state.

Directory structure

src/
  domain/
  application/
  infrastructure/
  plugins/
  prompts/
  ui/
electron/
resources/
tests/
docs/

How to run development

1. Install dependencies:

npm install

2. Start the desktop app:

npm run dev

The overlay window opens as an always-on-top Electron window. The default shortcut is Ctrl+Shift+Space.

In-app help and docs

The app now exposes local documentation in two places, and both open inside a built-in read-only documentation window:

• the Help & Docs panel inside the overlay UI
• the tray menu under Documentation

Available in-app end-user documents:

• English Guide
• Chinese Guide
• German Guide

How to run tests

npm run test

How to build production assets

npm run build

This produces dist/ and dist-electron/.

How to generate an installer

npm run dist

Installer output is written to release/.

Windows/macOS/Linux distribution

• Windows target: nsis
• macOS target: dmg
• Linux target: AppImage

Current implementation-time verification priority is Windows. macOS and Linux packaging targets are configured but not validated to the same level in this MVP.

Build metadata and version display

• App name: Software Coach
• Version: 0.1.0
• App ID: com.softwarecoach.desktop
• Windows installer name: Software Coach Setup 0.1.0.exe
• Installer output path: release/Software Coach Setup 0.1.0.exe

Provider configuration

Default example:

models:
  default:
    provider: openai-compatible
    baseUrl: https://api.openai.com/v1
    model: gpt-4.1

  local:
    provider: openai-compatible
    baseUrl: http://localhost:11434/v1
    model: qwen2.5vl:7b

routing:
  vision:
    use: default
  text:
    use: local
  embeddings:
    use: local

privacy:
  allowCloudModels: false
  storeScreenshots: false
  telemetry: false
  allowRemoteDocs: false

API keys are never stored in config files. Use environment variables such as OPENAI_API_KEY.

How to configure Ollama

As of May 16, 2026, the app connects to Ollama through Ollama's OpenAI-compatible API at http://localhost:11434/v1. Ollama's official docs show base_url='http://localhost:11434/v1/', an ignored placeholder api_key='ollama', and a vision example on /v1/chat/completions. In this project, no real API key is needed for local Ollama because local base URLs bypass the API key requirement.

Example:

models:
  local:
    provider: openai-compatible
    baseUrl: http://localhost:11434/v1
    model: qwen2.5vl:7b
routing:
  vision:
    use: local
  text:
    use: local

If you prefer a named internal route, the code also supports provider: ollama.

Recommended local setup:

ollama pull qwen2.5vl:7b
ollama serve

Then in Software Coach:

• open Settings
• set Text Provider to ollama
• set Ollama Base URL to http://localhost:11434/v1
• set Local Model Name to gemma3:4b or qwen2.5vl:7b
• click Apply Settings

The app now persists these settings to a local config.yaml under the app data directory, so they survive restart.

Current settings behavior:

• when ollama is selected, the settings panel shows Ollama-specific fields only
• the app lists locally installed Ollama models from ollama list
• you can still type a custom model name manually
• if the custom model is not installed, the app asks for confirmation before running ollama pull <model>
• the settings panel shows installation progress while the model is being pulled

Best Ollama models for this task

As of May 16, 2026, the best Ollama local model for this project is qwen2.5vl:7b.

Reasoning:

• Ollama describes Qwen2.5-VL as its "new flagship vision-language model"
• the official page says it is highly capable of analyzing text, charts, icons, graphics, and layouts inside images
• that is exactly the workload here: reading software screenshots, table headers, buttons, metrics, and panel layouts
• the same page says Qwen2.5-VL-7B outperforms GPT-4o-mini on a number of tasks, which is a strong signal for a local screenshot-coaching use case

Recommended choices:

• qwen2.5vl:7b Best overall fit for this app. Recommended default for local screenshot coaching.
• gemma3:12b Best balanced fallback if you want a larger general multimodal model. Official Ollama library lists it as multimodal with a 128K context window. Good choice if you have more VRAM and want a broader reasoning model.
• gemma3:4b Good lower-resource fallback when qwen2.5vl:7b is too heavy.
• llama3.2-vision:11b Usable, but not my first choice here. Ollama's page explicitly notes that image+text applications are officially English-only, which is a worse fit for a product that may coach in Chinese, English, or German.
• minicpm-v Lightweight fallback for constrained machines, but not the first recommendation for high-accuracy software UI coaching.

Practical recommendation by machine size:

• 8 GB class local setup: start with gemma3:4b
• 12 to 16 GB class local setup: start with qwen2.5vl:7b
• 24 GB+ class local setup: try gemma3:12b and compare against qwen2.5vl:7b

Official sources:

• Ollama OpenAI compatibility
• Ollama Qwen2.5-VL library page
• Ollama Gemma 3 library page
• Ollama Llama 3.2 Vision library page
• Ollama MiniCPM-V library page

How to configure an OpenAI-compatible provider

Example cloud route:

models:
  default:
    provider: openai-compatible
    baseUrl: https://api.openai.com/v1
    model: gpt-4.1

Set:

set OPENAI_API_KEY=your_key_here

If privacy.allowCloudModels is false, cloud routes fall back to the mock provider.

How to create a new Coach Pack

1. Add a new folder under src/plugins/<software-id>/
2. Create:
   • manifest.yaml
   • detector.ts
   • pack.ts
   • docs/
   • workflows/
   • ontology/
   • prompts/
   • fixtures/
   • optional state-graph/
   • optional trace/
3. Implement the CoachPack interface.
4. Register the pack in src/application/createServices.ts.

Core application orchestration should not need modification.

How to add a new ModelProvider

1. Create a provider in src/infrastructure/models/
2. Implement the ModelProvider interface
3. Add its routing logic in modelProviderFactory.ts
4. Express its differences through capabilities

PerfView MVP can do

• detect likely PerfView screens from direct vision analysis and pack-grounded hints
• explain common views such as CPU Stacks, CallTree, ByName, GC Heap, Thread Time, and Events
• answer CPU-high, GC-heap, memory-leak, and Thread-Time style questions using local docs
• provide next-step guidance without clicking
• expose debug details and citations
• support progressive actions such as I completed this step and Still stuck
• hide Software Coach, let the user drag a region on the primary display, and save only the cropped area through Electron main process

Current limitations

• no ETL or trace parsing yet
• no automatic clicking or overlay highlighting
• screenshot capture currently supports drag-to-select region capture on the primary display only
• the best screenshot understanding currently depends on a vision-capable provider
• when software context is not confirmed, the app will ask clarifying questions before detailed coaching
• manual image selection remains as a fallback screenshot path for MVP
• non-PerfView software options are UI-level placeholders for future packs
• settings now persist locally and support provider-aware fields, but they are still an MVP settings surface
• the current packaged Electron runtime uses sandbox: false as a temporary preload compatibility workaround
• macOS and Linux targets are configured, but Windows is the only validated packaging path today

Automatic updates

electron-updater is wired as a non-blocking placeholder. The current app starts normally without waiting on update checks.

Future update strategies:

• GitHub Releases
• internal enterprise update server
• signed private distribution channels

Detailed implementation and release notes

For the complete Windows-first delivery status, troubleshooting, implemented features, extension points, and TODO inventory, see docs/windows-first-delivery.md.

Future roadmap

• restore sandbox: true for packaged builds after preload compatibility is fixed cleanly
• extend region capture to multi-display selection
• add OCR providers
• add vector retrieval
• add trace analysis
• add overlay highlighting
• add workflow verification
• add enterprise mode
• add fully local-only hardened mode
• add richer UI state graph reasoning