Skip to content

test(e2e): add insomnia-test Playwright framework, CI workflow and skill [1/6] - #10594

Open
bobqiu-kong wants to merge 18 commits into
developfrom
test/e2e-1-framework
Open

bobqiu-kong wants to merge 18 commits into
developfrom
test/e2e-1-framework

Conversation

@bobqiu-kong

@bobqiu-kong bobqiu-kong commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Introduces packages/insomnia-test, a Playwright + Electron E2E framework with a strict layered architecture, plus the sharded Insomnia Test GitHub Actions workflow and the write-insomnia-test Claude skill. Ships with a single smoke spec (http-request/happy-path) proving the stack end to end; all other specs follow in the stacked PRs below.

insomnia-smoke-test is not modified.

Stack

Part of a stacked E2E suite. Merge in order:

  1. fix(project-modal): close modal on redirect by comparing pathname #10593 fix/project-modal-close-on-redirect — app fix (independent, based on develop)
  2. test(e2e): add insomnia-test Playwright framework, CI workflow and skill [1/6] #10594 test/e2e-1-framework — framework + CI + skill
  3. test(e2e): add HTTP, cookie, environment and template-tag specs [2/6] #10595 test/e2e-2-http-cookie-env — HTTP, cookie, environment, template-tag
  4. test(e2e): add GraphQL, gRPC, WebSocket, Socket.IO, SSE and MCP specs [3/6] #10596 test/e2e-3-protocols — GraphQL, gRPC, WebSocket, Socket.IO, SSE, MCP
  5. test(e2e): add workspace, document, runner, mock-server, app and preferences specs [4/6] #10597 test/e2e-4-workspace — workspace, document, runner, mock-server, app, preferences
  6. test(e2e): add git-sync, cloud-sync, organization and konnect specs [5/6] #10598 test/e2e-5-sync — git-sync, cloud-sync, organization, konnect
  7. test(e2e): add data, migration and plugin specs [6/6] #10599 test/e2e-6-data — data, migration, plugin

Why

The existing smoke-test package is hard to scale: specs are coupled to selectors, helpers are ad hoc, and CI runs are slow and hard to debug. This builds a replacement foundation that is easier to write, review and scale horizontally. The workflow is not yet a required check, so it can stabilise alongside the existing suite before any cut-over.

What's changed

1. Layered package packages/insomnia-test

tests/*.spec.ts  →  flows/*.flow.ts  →  pages/*.page.ts  →  models/*.ts (+ enums/*.ts)
Layer Purpose
tests/ Scenario specs by feature area (only the smoke spec in this PR)
flows/ Multi-step user journeys, registered in flow-manager.ts
pages/ Page objects holding selectors/interactions, registered in page-manager.ts
models/, enums/ Typed inputs and shared constants
misc/ Electron fixtures/launcher, coverage, step instrumentation, throwOnDialog decorator, TLS cert generation, base mock servers

All flows/pages/models land here (not split per feature) because they are mutually dependent — e.g. WorkspaceFlow creates requests of every protocol — so they can't be separated without breaking type-checks. The follow-up PRs are specs + their mock servers only.

Mock servers in this PR: Insomnia mock API (:4010) and HTTP echo (:4060, which loads misc/fixtures/petstore-swagger.json at startup, so that fixture lives here too). Others are added by the PRs that need them.

Launch modes (misc/fixtures.ts, playwright.config.ts): dev mode (default locally) launches Electron from packages/insomnia against the Vite dev server (:3334); setting INSOMNIA_BINARY (CI always does) tests the packaged build; INSOMNIA_DEV_MODE=false uses the installed app.

Coverage & diagnostics: opt-in V8 coverage (COVERAGE=true) via monocart-coverage-reports with per-shard raw data and merge scripts; trace/screenshot/video retained on failure.

2. CI: .github/workflows/insomnia-test.yml

Runs on pushes to develop and PRs, with per-ref concurrency cancellation.

  1. build — builds the Linux unpacked binary once, cached by commit SHA. A build failure fails the run (shards need the cached binary).
  2. parallel-test — sharded matrix, each shard restores the cached binary and runs under xvfb (retries: 2 on CI); uploads raw coverage, HTML and blob reports.
  3. insomnia-test-result — merges coverage and blob reports into coverage-report / playwright-report; every step is continue-on-error.

3. Supporting changes

  • packages/insomnia-data: add require conditions to exports (CommonJS test package imports it); export GCPCredential type.
  • packages/insomnia/electron-builder.config.js: keep .map files when KEEP_SOURCEMAPS=true so coverage maps to source.
  • Root package.json / package-lock.json: register the workspace. eslint.config.mjs: Playwright rules for the package, Node globals for standalone CJS mock servers. .gitignore: ignore reports/generated fixtures.
  • .claude/skills/write-insomnia-test/: skill for writing new specs from plain-language scenarios using the layers above.

Test plan

  • npx playwright test --list in packages/insomnia-test lists the smoke spec (verified locally)
  • Insomnia Test workflow: build succeeds and the shard running happy-path passes
  • Insomnia Test Result publishes merged report artifacts

Notes for reviewers

  • ~23.7k added lines, nearly all mechanical page/flow/model code; suggested review order: misc/fixtures.ts + launch.ts → flows/base.flow.ts / pages/base.page.ts → CI workflow.
  • The workflow does not yet make this a required check; retiring insomnia-smoke-test is a follow-up.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

✅ KSAI Review: Finished

  1. 09:14 UTC Reviewing the patch and checking related tests and types
  2. 09:16 UTC Reviewing the change diff and tracing related code paths
  3. 09:18 UTC Reviewing patch hunks and tracing referenced test fixtures
  4. 09:20 UTC Reading the patch and cross-checking changed files for issues
  5. 09:25 UTC Reviewing diff hunks and tracing cross-package references

Run report (federated) · 1 paid run · $0.8311 total
# Engine Result Model Turns Cost
1 opencode success glm-5.3/high 50 $0.8305

Reviewing cost $0.8305; deciding how to run it cost $0.0006

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check Result
Scope New packages/insomnia-test Playwright E2E suite (flows, pages, models, mock servers), insomnia-test CI workflow, eslint/package.json wiring, plus small insomnia-data and electron-builder.config.js edits.
Mandate typescript-code-reviewer
Findings 1 High, 1 Medium, 5 Low
Findings audit Ran kreview:findings-auditor. All 7 findings upheld; 2 line locations corrected against the files on disk.

The E2E framework itself is solid, but it cannot run: echo-server reads a fixture file this PR never commits, so the webServer never starts and every Playwright run aborts. Most flows also target mock servers absent from the repo, and the CI comment about build failures not blocking the PR is false. Fix these before merging.

Additional Risk pass: no hardcoded secrets found; mock keys are generated per process. Unauthenticated _admin mock endpoints listen on all interfaces, and per-test data directories are never deleted. Both reported as findings.

This review read b94e532. The pull request has moved to 86da056 since, so a comment on a line changed after that commit shows as outdated.

Comment thread packages/insomnia-test/misc/echo-server.js
Comment thread packages/insomnia-test/misc/git-fixtures.ts
Comment thread .github/workflows/insomnia-test.yml Outdated
Comment thread packages/insomnia-test/misc/mock-api.js Outdated
Comment thread packages/insomnia-test/misc/fixtures.ts
Comment thread packages/insomnia-test/pages/base.page.ts
Comment thread packages/insomnia-test/flows/cookie.flow.ts
@cwangsmv
cwangsmv added this pull request to stack #10600 October 13, 2026 09:52
Comment thread packages/insomnia-test/misc/mock-bin-server.js Fixed
@bobqiu-kong
bobqiu-kong force-pushed the test/e2e-1-framework branch 2 times, most recently from 32503d2 to 386cee6 Compare October 14, 2026 03:06
@cwangsmv
cwangsmv requested review from amy-kong-hq and a balanced review from Copilot October 14, 2026 05:21
Comment thread packages/insomnia-test/tests/http-request/happy-path.spec.ts Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The framework currently has type-check, dependency-lock, lifecycle-cleanup, listener, connection, error-handling, and workflow-rollout defects.

7 open findings
What changed in this PR

Introduces a layered Playwright/Electron E2E framework, supporting mock services and diagnostics, plus sharded GitHub Actions execution.

Changes:

  • Adds reusable models, pages, flows, fixtures, mock servers, and an HTTP smoke test.
  • Adds build caching, test sharding, coverage merging, and report publication.
  • Adds test-package configuration, documentation, and source-map packaging support.
File Description
.claude/​skills/​write-insomnia-test/​SKILL.md Defines E2E authoring skill.
.claude/​skills/​write-insomnia-test/​reference.md Documents framework APIs.
.github/​workflows/​insomnia-test.yml Adds sharded E2E workflow.
.gitignore Ignores generated test artifacts.
eslint.config.mjs Configures test-package linting.
package-lock.json Locks new workspace dependencies.
package.json Registers the test workspace.
packages/​insomnia-data/​package.json Adds CommonJS export conditions.
packages/​insomnia-data/​src/​models/​types.ts Exports gRPC credential type.
packages/​insomnia-test/​enums/​api-spec-export-format.ts Defines API export formats.
packages/​insomnia-test/​enums/​auth-type.ts Defines authentication types.
packages/​insomnia-test/​enums/​collaborator-role.ts Defines collaborator roles.
packages/​insomnia-test/​enums/​content-type.ts Defines request content types.
packages/​insomnia-test/​enums/​context-menu-items.ts Defines context-menu actions.
packages/​insomnia-test/​enums/​delete-mode.ts Defines deletion modes.
packages/​insomnia-test/​enums/​export-format.ts Defines export formats.
packages/​insomnia-test/​enums/​http-method.ts Defines HTTP methods.
packages/​insomnia-test/​enums/​import-sources.ts Defines import sources.
packages/​insomnia-test/​enums/​lint-severity.ts Defines lint severities.
packages/​insomnia-test/​enums/​mock-body-type.ts Defines mock body types.
packages/​insomnia-test/​enums/​project-types.ts Defines project types.
packages/​insomnia-test/​enums/​request-state.ts Defines request states.
packages/​insomnia-test/​enums/​ruleset-type.ts Defines ruleset types.
packages/​insomnia-test/​enums/​script-sandbox-rule-group.ts Defines sandbox rule groups.
packages/​insomnia-test/​enums/​script-tab.ts Defines script tabs.
packages/​insomnia-test/​enums/​spec-format.ts Defines specification formats.
packages/​insomnia-test/​enums/​sync-status.ts Defines sync states.
packages/​insomnia-test/​enums/​tree-node-types.ts Defines workspace node types.
packages/​insomnia-test/​flows/​app.flow.ts Orchestrates app lifecycle.
packages/​insomnia-test/​flows/​base.flow.ts Provides shared flow behavior.
packages/​insomnia-test/​flows/​certificates.flow.ts Orchestrates certificate actions.
packages/​insomnia-test/​flows/​cloud-sync.flow.ts Orchestrates cloud synchronization.
packages/​insomnia-test/​flows/​cookie.flow.ts Orchestrates cookie actions.
packages/​insomnia-test/​flows/​environment.flow.ts Orchestrates environment actions.
packages/​insomnia-test/​flows/​event-stream-request.flow.ts Orchestrates SSE requests.
packages/​insomnia-test/​flows/​export.flow.ts Orchestrates data exports.
packages/​insomnia-test/​flows/​flow-context.ts Defines shared flow context.
packages/​insomnia-test/​flows/​flow-manager.ts Registers flow instances.
packages/​insomnia-test/​flows/​folder.flow.ts Orchestrates folder actions.
packages/​insomnia-test/​flows/​git-sync.flow.ts Orchestrates Git synchronization.
packages/​insomnia-test/​flows/​graphql-request.flow.ts Orchestrates GraphQL requests.
packages/​insomnia-test/​flows/​grpc-request.flow.ts Orchestrates gRPC requests.
packages/​insomnia-test/​flows/​http-request.flow.ts Orchestrates HTTP requests.
packages/​insomnia-test/​flows/​import.flow.ts Orchestrates data imports.
packages/​insomnia-test/​flows/​mcp-client.flow.ts Orchestrates MCP clients.
packages/​insomnia-test/​flows/​mock-server.flow.ts Orchestrates mock servers.
packages/​insomnia-test/​flows/​organization.flow.ts Orchestrates organization actions.
packages/​insomnia-test/​flows/​preferences.flow.ts Orchestrates preference changes.
packages/​insomnia-test/​flows/​socket-io-request.flow.ts Orchestrates Socket.IO requests.
packages/​insomnia-test/​flows/​template-tag.flow.ts Orchestrates template tags.
packages/​insomnia-test/​flows/​web-socket-request.flow.ts Orchestrates WebSocket requests.
packages/​insomnia-test/​flows/​workspace.flow.ts Orchestrates workspace actions.
packages/​insomnia-test/​misc/​certs.js Generates TLS test certificates.
packages/​insomnia-test/​misc/​constants.ts Defines shared test constants.
packages/​insomnia-test/​misc/​coverage-global-setup.ts Initializes coverage collection.
packages/​insomnia-test/​misc/​coverage-global-teardown.ts Finalizes coverage collection.
packages/​insomnia-test/​misc/​coverage.ts Collects V8 coverage.
packages/​insomnia-test/​misc/​decorators.ts Adds dialog-error handling.
packages/​insomnia-test/​misc/​echo-server.js Provides HTTP echo service.
packages/​insomnia-test/​misc/​event-stream-server.js Provides SSE test service.
packages/​insomnia-test/​misc/​export-file.ts Supports export assertions.
packages/​insomnia-test/​misc/​fixtures.ts Defines Electron test fixtures.
packages/​insomnia-test/​misc/​fixtures/​petstore-swagger.json Supplies OpenAPI fixture data.
packages/​insomnia-test/​misc/​fixtures/​upload-sample.txt Supplies upload fixture data.
packages/​insomnia-test/​misc/​git-fixtures.ts Defines Git test fixtures.
packages/​insomnia-test/​misc/​git-server.js Provides Git HTTP service.
packages/​insomnia-test/​misc/​grpc-server.js Provides gRPC test service.
packages/​insomnia-test/​misc/​launch.ts Launches tested Electron builds.
packages/​insomnia-test/​misc/​mcp-server.js Provides MCP test service.
packages/​insomnia-test/​misc/​merge-coverage-union.js Unions shard coverage.
packages/​insomnia-test/​misc/​merge-coverage.js Generates merged coverage reports.
packages/​insomnia-test/​misc/​mock-api.js Mocks Insomnia APIs.
packages/​insomnia-test/​misc/​mock-bin-server.js Provides mock-bin service.
packages/​insomnia-test/​misc/​oauth2-server.js Provides OAuth2 service.
packages/​insomnia-test/​misc/​protos/​hello.proto Defines test gRPC service.
packages/​insomnia-test/​misc/​socket-server.js Provides Socket.IO service.
packages/​insomnia-test/​misc/​step-instrumentation.ts Instruments page and flow steps.
packages/​insomnia-test/​misc/​ws-echo-server.js Provides WebSocket echo service.
packages/​insomnia-test/​models/​certificate.ts Models certificates.
packages/​insomnia-test/​models/​collection.ts Models collections.
packages/​insomnia-test/​models/​commit.ts Models Git commits.
packages/​insomnia-test/​models/​cookie.ts Models cookies.
packages/​insomnia-test/​models/​curl-command.ts Models cURL commands.
packages/​insomnia-test/​models/​environment.ts Models environments.
packages/​insomnia-test/​models/​event-stream-request.ts Models SSE requests.
packages/​insomnia-test/​models/​folder.ts Models folders.
packages/​insomnia-test/​models/​git-credential.ts Models Git credentials.
packages/​insomnia-test/​models/​git-repo-connection.ts Models Git connections.
packages/​insomnia-test/​models/​graphql-request.ts Models GraphQL requests.
packages/​insomnia-test/​models/​grpc-request.ts Models gRPC requests.
packages/​insomnia-test/​models/​http-request.ts Models HTTP requests.
packages/​insomnia-test/​models/​mcp-client.ts Models MCP clients.
packages/​insomnia-test/​models/​mock-route.ts Models mock routes.
packages/​insomnia-test/​models/​mock-server.ts Models mock servers.
packages/​insomnia-test/​models/​project.ts Models projects.
packages/​insomnia-test/​models/​response.ts Models response data.
packages/​insomnia-test/​models/​runner.ts Models runner results.
packages/​insomnia-test/​models/​settings.ts Models application settings.
packages/​insomnia-test/​models/​socket-io-request.ts Models Socket.IO requests.
packages/​insomnia-test/​models/​template-tag.ts Models template tags.
packages/​insomnia-test/​models/​websocket-request.ts Models WebSocket requests.
packages/​insomnia-test/​package.json Configures the test package.
packages/​insomnia-test/​pages/​auth-tab.page.ts Automates authentication fields.
packages/​insomnia-test/​pages/​base.page.ts Provides shared page behavior.
packages/​insomnia-test/​pages/​certificates.page.ts Automates certificate settings.
packages/​insomnia-test/​pages/​cloud-sync.page.ts Automates cloud sync.
packages/​insomnia-test/​pages/​command-palette.page.ts Automates command palette actions.
packages/​insomnia-test/​pages/​cookie.page.ts Automates cookie management.
packages/​insomnia-test/​pages/​environment.page.ts Automates environment editing.
packages/​insomnia-test/​pages/​event-stream-request.page.ts Automates SSE request UI.
packages/​insomnia-test/​pages/​export.page.ts Automates export UI.
packages/​insomnia-test/​pages/​folder.page.ts Automates folder UI.
packages/​insomnia-test/​pages/​git-sync.page.ts Automates Git sync UI.
packages/​insomnia-test/​pages/​graphql-request.page.ts Automates GraphQL request UI.
packages/​insomnia-test/​pages/​grpc-request.page.ts Automates gRPC request UI.
packages/​insomnia-test/​pages/​http-request.page.ts Automates HTTP request UI.
packages/​insomnia-test/​pages/​import.page.ts Automates import UI.
packages/​insomnia-test/​pages/​invite.page.ts Automates collaborator invites.
packages/​insomnia-test/​pages/​konnect.page.ts Automates Konnect UI.
packages/​insomnia-test/​pages/​mcp-client.page.ts Automates MCP client UI.
packages/​insomnia-test/​pages/​mock-server.page.ts Automates mock-server UI.
packages/​insomnia-test/​pages/​page-manager.ts Registers page objects.
packages/​insomnia-test/​pages/​preferences.page.ts Automates preferences UI.
packages/​insomnia-test/​pages/​project-settings.page.ts Automates project settings.
packages/​insomnia-test/​pages/​request.page.ts Provides shared request actions.
packages/​insomnia-test/​pages/​response.page.ts Reads response state.
packages/​insomnia-test/​pages/​runner.page.ts Automates collection runner UI.
packages/​insomnia-test/​pages/​script-tab.page.ts Automates script editors.
packages/​insomnia-test/​pages/​socket-io-request.page.ts Automates Socket.IO UI.
packages/​insomnia-test/​pages/​tab-panel.page.ts Provides shared tab behavior.
packages/​insomnia-test/​pages/​template-tag.page.ts Automates template-tag UI.
packages/​insomnia-test/​pages/​web-socket-request.page.ts Automates WebSocket UI.
packages/​insomnia-test/​pages/​workspace.page.ts Automates workspace navigation.
packages/​insomnia-test/​playwright.config.ts Configures Playwright and services.
packages/​insomnia-test/​tests/​http-request/​happy-path.spec.ts Adds end-to-end HTTP smoke coverage.
packages/​insomnia/​electron-builder.config.js Retains source maps for coverage.

🧠 Review effort: Balanced

Comment thread packages/insomnia-test/flows/http-request.flow.ts Outdated
Comment thread packages/insomnia-test/package.json
Comment thread packages/insomnia-test/pages/event-stream-request.page.ts
Comment thread packages/insomnia-test/pages/response.page.ts
Comment thread .github/workflows/insomnia-test.yml
Comment thread packages/insomnia-test/flows/app.flow.ts
Comment thread packages/insomnia-test/pages/request.page.ts
Comment thread packages/insomnia-test/flows/http-request.flow.ts Outdated
Comment thread packages/insomnia-test/flows/workspace.flow.ts Outdated
Comment thread .vscode/settings.json Outdated
Comment thread .gitignore Outdated
Comment thread packages/insomnia-test/pages/web-socket-request.page.ts
Comment thread packages/insomnia-test/pages/base.page.ts
@bobqiu-kong
bobqiu-kong force-pushed the test/e2e-1-framework branch 7 times, most recently from 2c352ce to ccad0e3 Compare October 19, 2026 09:52
…ite-insomnia-test skill

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
bobqiu-kong and others added 17 commits October 20, 2026 09:33
… at startup

misc/echo-server.js reads misc/fixtures/petstore-swagger.json when it
starts, so the fixture must land together with the server; otherwise every
shard's webServer exits with ENOENT.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- bind mock-api and echo-server to 127.0.0.1
- remove per-test dataPath after use
- throw when setCodeMirrorValue does not hold after retries
- drop continue-on-error from the build job
- shorten cookie waitForPersisted docstring

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds the event-stream, socket.io, ws-echo, mcp, grpc, git, oauth2 and
mock-bin servers (bound to 127.0.0.1), their proto/upload fixtures, npm
scripts and Playwright webServer entries, so flows that reference them
no longer fail with connection refused.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… nosniff

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…response

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
WorkspaceFlow now takes a structural RequestFlows dependency instead of
importing FlowManager, which removes the 3 new circular references.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The type represents the response pane (response data plus console, stream
events and script test results), not the network response, and clashed
with the insomnia-data Response name. Update flows, pages, specs, JSDoc
and the write-insomnia-test skill accordingly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ones

WorkspaceFlow.create/duplicate/rename, ExportFlow.export and
EnvironmentFlow.create dispatched on argument types via typeof/instanceof
and casts, so a call site did not show which flow it triggered. Replace
them with explicitly named methods (createProject, openLocalGitFolder,
cloneGitProject, createCollection, duplicateRequest, renameCollection,
exportProject, createBaseEnvironment, ...) and update the
write-insomnia-test skill docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drop the pass-through re-exports of expect from misc/fixtures.ts and
misc/git-fixtures.ts, import it from @playwright/test in specs, and
update the write-insomnia-test skill accordingly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants