Skip to content

feat(ocr): make document parsing configurable, add hosted MinerU/PaddleOCR support - #2672

Open
wachoo wants to merge 2 commits into
dataelement:mainfrom
wachoo:pr/ocr-upstream-main
Open

wachoo wants to merge 2 commits into
dataelement:mainfrom
wachoo:pr/ocr-upstream-main

Conversation

@wachoo

@wachoo wachoo commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

背景

文档解析(OCR/ETL)此前只能手改存储的 YAML,且 PaddleOCR loader 无法对接托管的 jobs API。

改动

  • PaddleOCR:新增 jobs-API 传输(multipart 提交 → 轮询 jobId → 取预签名结果),URL 以 /jobs 结尾时自动选用;自建 /layout-parsing 路径保持不变
  • MinerU V4:支持托管签名上传接口(/api/v1/agent/parse/file,无需公网地址)与公网 URL 接口(/api/v4/extract/task);自建 /v1 客户端不变
  • MinerU:加密 API Key 按可配置请求头注入;request_kwargs.backend 可编辑
  • 凭据优先取 DB 加密值,其次环境变量;任何接口都不回传令牌
  • 配置写入为文本级合并:保留注释;旧环境缺失的子段自动创建;内容未变则不写库
  • 新增管理端「系统管理 → 文档解析」表单:按 provider 提示 + 连接测试(区分令牌被拒/服务不可达,且不消耗解析额度)
  • 传输层按地址形态自动选择,三种部署形态互不影响

与完整分支的差异

本 PR 只含该功能提交,基线为上游 main。原提交中涉及 Makefile / deploy.md 的改动已被丢弃——这两个文件在上游不存在(由另一条开发栈提交引入)。因此本 PR 不引入 docker/.env.app 的本地开发注入机制;compose / 环境变量方式不受影响。

验证

  • 新增/相关后端测试 74 passed(配置文本更新、schema、令牌解析、探活、两个 hosted 客户端)
  • 前端:TypeScript 无错误、ESLint 通过、check-i18n 无新增漂移
  • 真实服务端到端:上传 → Celery → parse_type=paddle_ocr / mineru_v4,文件状态 SUCCESS
  • 本提交自身的 ruff 检查干净(仅剩的文件告警为上游既有)

已知限制

  • MinerU 签名上传接口只返回 Markdown,没有版面坐标,引用高亮不可用;需要坐标请用 /api/v4/extract/task(文件需公网可达)
  • 加密使用仓库内置 Fernet key(在 git 中),属混淆而非保密

Document-parse providers were configurable only by hand-editing the stored
YAML, and the PaddleOCR loader could not talk to the hosted jobs API at all.

- PaddleOCR: add a jobs-API transport (multipart submit → poll jobId → fetch
  the presigned result), selected automatically when the URL ends in /jobs.
  The self-hosted /layout-parsing path is unchanged.
- MinerU V4: support the hosted upload API (/api/v1/agent/parse/file) and the
  public-URL API (/api/v4/extract/task); the self-hosted /v1 job client is
  untouched. Upload mode returns markdown only, so it has no bbox metadata.
- MinerU: the encrypted API key is injected as a configurable request header,
  and request_kwargs.backend is editable.
- Providers resolve credentials from an encrypted DB value first, then the
  environment; no token is ever returned by the API.
- Config writes are text-level and preserve comments; a nested section missing
  from an older deployment is created instead of failing the save.
- Admin form under 系统管理 → 文档解析 with per-provider guidance and a
  connection test that distinguishes a rejected token from an unreachable
  service without consuming a parse job.
- Native dev sources docker/.env.app (git-ignored); compose passes the token
  variables through. No secret is committed.
PR 2672 touches docker/docker-compose.yml, which triggers App Runtime
Quality. That gate tests src/runtime-manager and src/app-proxy, neither of
which this PR changes — both suites were already red on main:

- F114 stage permits: the executor doubles lack authorize_step/preflight.
- FastAPI 0.141 keeps included routers lazy, so app.routes no longer
  exposes .path; assert the public HTTP contract instead.
- The image-check helper exists in the templates but was missing from the
  log-container whitelist and from the pod-count/fixture expectations.
- Helm 3.16 server-side dry run maps every rendered resource, so the
  lookup fixture must advertise the chart's API resources.
- app-proxy: the trailing-slash redirect is a deliberate second
  authorization; the fail-closed cache test must use the canonical URL.

Test-only except the one-line k8s_client log whitelist. Verified locally:
runtime-manager 1861 passed / 90 skipped, app-proxy 273 passed, ruff clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant