Skip to content

feat: add authenticated MCP remote control - #831

Merged
imguoguo merged 4 commits into
sipeed:mainfrom
SiYue-ZO:feature/mcp
Jul 29, 2026
Merged

feat: add authenticated MCP remote control#831
imguoguo merged 4 commits into
sipeed:mainfrom
SiYue-ZO:feature/mcp

Conversation

@SiYue-ZO

@SiYue-ZO SiYue-ZO commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

This PR adds authenticated MCP remote control support to NanoKVM and integrates it with PicoClaw so external automation, PicoClaw AI control, and local user input can safely share the same HID/control surface.

The implementation introduces a backend MCP server, shared control ownership management, safer input serialization, PicoClaw runtime handoff behavior, and the corresponding web UI/settings/i18n updates.

Motivation

NanoKVM currently exposes local UI and PicoClaw-driven control flows, but there was no structured way for an external MCP client to operate the device safely. Adding MCP support requires more than exposing keyboard and mouse APIs: remote control must be authenticated, mutually coordinated with PicoClaw, and protected against conflicting input sources.

This PR adds that coordination layer so control ownership is explicit, observable, and released cleanly.

Backend Changes

  • Add an authenticated MCP Streamable HTTP server.
  • Add MCP tools for:
    • Screenshot capture
    • Keyboard input
    • Mouse movement/click/scroll actions
  • Add MCP configuration and service lifecycle handling.
  • Add dedicated MCP API routes and protocol types.
  • Introduce shared controlmode management for ownership state.
  • Introduce inputcontrol coordination to serialize input activity across MCP, PicoClaw, and local HID paths.
  • Update HID services with queued/released input behavior to avoid stale key or mouse state after control changes.
  • Add PicoClaw runtime control operations for takeover, release, interruption, and status reporting.
  • Improve PicoClaw gateway startup/stop behavior with PID-file based process tracking.
  • Update server Go version requirement and dependencies required by MCP support.

Web Changes

  • Add a new MCP settings page in the desktop settings menu.
  • Add frontend API bindings for MCP configuration.
  • Add AI-control ownership state to the frontend.
  • Update PicoClaw sidebar/runtime UI to reflect control ownership and operation state.
  • Prevent conflicting keyboard actions while AI/MCP control is active.
  • Preserve PicoClaw chat/session UI state when control is released.
  • Update related desktop input components so control handoff behavior is consistent.

Internationalization

  • Add locale coverage for MCP settings.
  • Add translations for PicoClaw control ownership, runtime operations, and conflict/status messages.
  • Update all existing locale files touched by the new UI strings.

@imguoguo
imguoguo requested a review from watermeko July 17, 2026 02:21
SiYue-ZO added 3 commits July 20, 2026 17:58
@SiYue-ZO
feat(mcp): add authenticated remote control backend
e3a204f 
Add an MCP Streamable HTTP server with token authentication, screenshot capture, keyboard, and mouse tools.

Introduce shared control-mode and input-control coordination so MCP, PicoClaw, and local HID paths serialize ownership safely.

Integrate PicoClaw gateway/runtime control handoff with PID-managed startup and focused unit coverage.
@SiYue-ZO
feat(web): add MCP settings and control handoff UI
89c5b4b 
Add the MCP settings view and frontend API bindings for configuring remote-control access.

Expose AI-control ownership state in PicoClaw, lock conflicting keyboard input paths, and preserve chat state when control is released.
@SiYue-ZO
i18n(picoclaw): add MCP control translations
138b4c7 
Add locale coverage for MCP settings, AI-control ownership, PicoClaw runtime operations, and related conflict states.
@SiYue-ZO
fix(picoclaw): cancel runtime lifecycle on mode switch
816ca18 
Cancel PicoClaw start/readiness waits when MCP takes control, clear held manual input state after failed release reports, and keep paste duration below the control-mode wait budget.

Also add MCP copy failure translations and avoid reconnect effect churn when the locale changes.
@watermeko
watermeko requested a review from Copilot July 23, 2026 03:58
Copilot AI reviewed Jul 23, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces authenticated MCP remote control support and integrates it with PicoClaw’s runtime/control-ownership model so external MCP clients, PicoClaw AI control, and local/manual input can safely share a single HID/control surface.

Changes:

  • Add MCP server endpoints, auth, configuration, and HID tooling (screenshot + keyboard/mouse actions).
  • Introduce/control “control_mode” ownership + operation cancellation/serialization across MCP, PicoClaw, and manual input.
  • Update web UI state management (keyboard lock coordination, reconnect flows, MCP settings, and i18n strings).

Reviewed changes

Copilot reviewed 132 out of 133 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
web/src/pages/desktop/picoclaw/sidebar-session-actions.ts Sidebar send/reconnect gating + refresh options
web/src/pages/desktop/picoclaw/sidebar-model-config.tsx Keyboard lock + disabled state handling
web/src/pages/desktop/picoclaw/sidebar-install.tsx Disable install action support
web/src/pages/desktop/picoclaw/sidebar-header.tsx Disable actions + runtime toggle title
web/src/pages/desktop/picoclaw/message-input.tsx Keyboard lock on focus + send result semantics
web/src/pages/desktop/picoclaw/keyboard-lock.ts Shared PicoClaw keyboard-lock helpers
web/src/pages/desktop/picoclaw/action-overlay.tsx Overlay visibility gating
web/src/pages/desktop/mouse/relative.tsx Release mouse state on cleanup
web/src/pages/desktop/menu/wol/index.tsx Replace global keyboard enable with lock sources
web/src/pages/desktop/menu/terminal/serial-port.tsx Keyboard lock in modal + formatting cleanup
web/src/pages/desktop/menu/settings/index.tsx Add MCP settings tab + keyboard lock
web/src/pages/desktop/menu/settings/device/advanced/autostart.tsx Keyboard lock for autostart settings
web/src/pages/desktop/menu/keyboard/shortcuts/recorder.tsx Keyboard lock while recording shortcuts
web/src/pages/desktop/menu/keyboard/leader-key.tsx Keyboard lock while recording leader key
web/src/lib/picoclaw-gateway.ts Control-mode events + close-code handling
web/src/lib/mouse.ts Absolute reset parameterization
web/src/jotai/picoclaw.ts Runtime status extended with control metadata
web/src/jotai/keyboard.ts Multi-source keyboard lock mechanism
web/src/jotai/ai-control.ts Normalize AI control status payloads
web/src/api/picoclaw.ts Add AI control status/mode endpoints
web/src/api/mcp.ts MCP config + key regeneration API bindings
web/src/i18n/locales/zh.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/zh_tw.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/vi.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/uk.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/tr.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/th.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/se.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/ru.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/pt_br.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/pl.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/nl.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/nb.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/ko.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/ja.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/it.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/id.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/hu.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/fr.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/es.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/en.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/de.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/da.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/cz.ts MCP/PicoClaw control-mode strings
web/src/i18n/locales/ca.ts MCP/PicoClaw control-mode strings
server/service/ws/types.go WS client tracks manual lease + queued HID reports
server/service/ws/client_test.go Tests for heartbeat/manual-lease release + cooldown
server/service/vm/jiggler/jiggler.go Jiggler uses control-mode + background coordinator
server/service/picoclaw/types.go Runtime status/control metadata + close codes
server/service/picoclaw/session_manager.go Add session snapshot support
server/service/picoclaw/session_lock.go Add ReleaseOwned to prevent stale releases
server/service/picoclaw/session_lock_test.go Tests for ReleaseOwned semantics
server/service/picoclaw/session_cleanup.go Conditional HID release on mode + owned lock
server/service/picoclaw/runtime_session.go Runtime session release flow update
server/service/picoclaw/runtime_session_test.go Ensure session release works in MCP mode
server/service/picoclaw/runtime_install.go Ensure dependencies on install paths
server/service/picoclaw/runtime_defaults.go Config versioning update
server/service/picoclaw/runtime_constants.go PID file constant introduced
server/service/picoclaw/mcp_handler.go MCP tool schema + structured control conflicts
server/service/picoclaw/mcp_handler_test.go MCP bootstrap + control transition tests
server/service/picoclaw/gateway_proxy.go Enforce control-mode gating + owned release
server/service/picoclaw/gateway_client.go Add gateway probe error detail
server/service/picoclaw/errors.go Add control-mode error codes + data payload support
server/service/picoclaw/control_operations.go Track/cancel control operations on mode switch
server/service/picoclaw/control_operations_test.go Tests for operation cancellation + wait bounds
server/service/picoclaw/control_events.go Broadcast control-mode changed over gateway
server/service/picoclaw/control_events_test.go Tests for control-mode event payload
server/service/picoclaw/config.go Refresh model metadata without owning control mode
server/service/picoclaw/config_files.go Robust home resolution + lazy config init
server/service/picoclaw/agent_profile.go Lifecycle locking + updated status return
server/service/mcp/screenshot.go MCP screenshot interfaces/types
server/service/mcp/remote_test.go HID remote actions release/cancel tests
server/service/mcp/mouse.go MCP mouse report building + bounds
server/service/mcp/mouse_test.go MCP mouse report tests
server/service/mcp/keyboard_test.go MCP keyboard report tests
server/service/mcp/config.go MCP config + API key generation/persistence
server/service/mcp/capture/kvm/kvm.go KVM snapshotter adapter
server/service/mcp/capture/capture.go Screenshot capture implementation + retry/timeout
server/service/mcp/auth.go Bearer API key middleware gated by mode
server/service/hid/status.go HID reset acquires manual control lease
server/service/hid/service.go HID service wired to control/coordinator
server/service/hid/release.go Central HID release helpers
server/service/hid/release_test.go Tests for HID release behavior
server/service/hid/queue.go Queued report wrapper for coordinated writes
server/service/hid/keyboard.go Keyboard queue supports coordinated execution + cleanup
server/service/hid/hid.go Validate report lengths + error-returning write APIs
server/service/hid/hid_test.go Tests for report validation + paste budget
server/service/controlmode/activity.go Bounded shared/exclusive activity gate
server/router/router.go Wire control + MCP routers with shared service instances
server/router/picoclaw.go Inject PicoClaw service into router
server/router/mcp.go MCP routes + handler wiring
server/README.md Document Go version requirement
server/README_ZH.md Document Go version requirement (ZH)
server/README_JA.md Document Go version requirement (JA)
server/proto/mcp.go MCP config request/response types
server/go.mod Bump Go version + MCP dependencies
kvmapp/system/init.d/S96picoclaw PID-file tracking + safer start/stop

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread server/router/mcp.go
Comment thread server/service/picoclaw/runtime_session.go
@imguoguo
imguoguo merged commit 57e1f6a into sipeed:main Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@watermeko watermeko Awaiting requested review from watermeko

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@SiYue-ZO @imguoguo