[Feature][Tracking] Native MongoDB Catalog #70251

Description

Track the implementation of a native, read-only MongoDB Catalog on Doris master, from client dependencies and the first single-cursor query to safe pushdown, parallel scans, and release qualification.

Design: Complete design PDF — Chinese, 49 pages, 16 chapters, four diagrams.

The PDF covers the architecture, FE/BE contracts, Schema management, BSON-to-Doris mapping, predicate semantics, split planning, consistency, resource ownership, SQL analytics, regression coverage, and the implementation roadmap. It is an RFC: proposed interfaces and settings are not claims of implemented functionality. The code-reading baseline is c5bfd6819f393bde37b73147abddc2f9a44641cb.

Related PRs: #70249 provides client dependencies and packaging; #70327 integrates the usable native Catalog/Reader. Their M0 integration and artifact checks passed; #70456 contains acceptance follow-up fixes and tests.

Progress — 2026-11-11

Dependency preparation #70249 and native Catalog/Reader/BSON implementation #70327 are merged into master. M0 integration and functional acceptance passed on Linux x86_64 with MongoDB 8.0.20 and matching FE/ASAN BE builds. The acceptance fix and additional tests are in #70456, pending review/merge; merge that follow-up before closing the master M0 gate.

Agreed scope: matching-version FE/BE deployment. Software upgrade/downgrade qualification, SRV validation, multi-architecture validation, and FE/BE capability negotiation are outside this issue's scope and are not acceptance blockers. The versioned scan descriptor and BE version/parameter validation remain part of D02.

Acceptance evidence: see the Testing section of #70456. Complete clean FE/ASAN BE build and zero-violation FE Checkstyle; 31 MongoDB connector FE tests, 51 external-catalog FE tests, 44 selected BSON/native-reader BE tests; both native SQL suites independently compared with 2 passed, 0 failed/fatal/skipped. The internal-table materialization has 2,081 rows and a zero two-way result difference. The opt-in real-driver fault runner verifies EOF/LIMIT/decode-error cleanup, injected getMore failure, socket/query deadlines, cancellation after an observed blocked getMore, concurrent reads, zero remaining source cursors, and successful recovery. Trusted-CA TLS succeeds; untrusted certificates and hostname mismatches fail with source errors. Source metadata confirms compression negotiation in both drivers; EXPLAIN and native Profile counters are verified.

Issues found and addressed in #70456: connector metadata authentication/transport errors were swallowed as missing tables; the shared lookup now preserves the typed source failure and retry behavior. The 2,081-row batch fixture also fit within master's default batch size of 8,160, so boundary queries now explicitly use 1,024 rows to exercise getMore.

The agreed M0 checks are complete on the acceptance branch. Master closure is pending merge of #70456. MongoDB 7 and broader source-topology/failover qualification, performance and release soak remain at their later roadmap gates; M1/M2/RC are not marked complete by this update.

Use cases and initial scope

Query MongoDB collections through SQL for filters, joins, aggregation, windows, JSON/array analysis, and pattern matching. Doris evaluates operators that are not safely pushed down. Repeated full scans and indexed text analytics can use an explicit import into a Doris internal table.

• FE: a MongoDB connector using the existing Connector SPI and Java Sync Driver for metadata, Schema discovery, and scan planning.
• BE: a native C++ Reader using libmongoc/libbson with RAII; BSON conversion belongs in DataTypeSerDe.
• Initial qualification targets: ordinary collections, standard views, and time-series collections on MongoDB 7/8, replica sets and mongos. Basic reads for all three object kinds are required for the initial release; parallel scans, snapshot reads, and pushdown are qualified per object kind.
• Read-only access. Writes, source DDL, CDC, and cross-cluster global snapshots are outside the initial release.

Initial-release object support

View and time-series basic reads are part of M0/M1 development and mandatory release acceptance. An unimplemented optimization must not defer the entire object kind. Schema/metadata privileges are documented separately from query privileges; unavailable identity/version metadata must not be fabricated.

Source constraints: standard views and permissions/snapshot isolation, view collation, and time-series limitations.

Work items

M0 — Minimal single-cursor path

• D00 — Client dependencies and build artifacts (W1; PR [chore](build) Add native MongoDB client dependencies #70249): validate pinned drivers and transitive dependencies, licenses, complete FE/BE builds, plugin packaging, TLS/SASL, and compression in the agreed deployment environment. Integration is delivered by [feature](mongodb) Add native catalog, reader and BSON SerDe #70327 and acceptance evidence is linked in [fix](catalog) Preserve connector discovery failures and qualify MongoDB M0 #70456.
• D01 — Catalog and Schema management (W1–W4): register the FE connector and distinguish collection/view/timeseries metadata. Map ordinary-collection validators; support explicit/document Schema for view output and time-series measurements; record view collation/available definition metadata and timeField/metaField. Retain source logical types, pin Schema/configuration versions, and define refresh/evolution. Do not reuse a base-collection validator as a view-output Schema, require a time-series validator, or treat a validator as proof about historical rows.
• D02 — Versioned query and single-Split contract (W1–W3; PR [feature](mongodb) Add native catalog, reader and BSON SerDe #70327): serialize stable object kind, namespace, available collection UUID, configuration/Schema versions, field bindings and read policy. Wire version-1 scan descriptors and single-Split RPC/dispatch; BE validates the descriptor version and required parameters. SQL filters and projection stay local in M0, and views retain their source-defined collation. Active cursors, driver sessions, and client objects remain local to their owners.
• D03 — Native BE Reader (W2–W4): integrate C++ pool leases, cursor/getMore, batching, deadlines, cancellation, error translation, ScanNode execution, and Profile metrics for ordinary collections, standard views, and time-series collections. Query each logical namespace directly: MongoDB evaluates view pipelines and unpacks time-series measurements. Do not bypass view access controls or read system.buckets directly. Release leases/cursors on every success and failure path.
• D04a — Basic BSON DataTypeSerDe support (W2–W4): implement initial scalar/Nullable conversions and ObjectId as 12 raw bytes in VARBINARY(12); preserve source type/subtype metadata and copy borrowed BSON bytes before their backing storage is released.

M0 gate: Catalog creation, object-kind discovery, and minimal SELECT work for ordinary collections, standard views, and time-series collections with one Split/cursor. Basic Schema/type, authentication, cancellation, error, cleanup, and build/package checks pass. Filters remain in Doris until their pushdown contracts are verified; M1 completes object-specific qualification.

M1 — Complete single-cursor queries

• D04b — Remaining first-release type mappings (W5–W6): complete the design's numeric/Decimal128, UTC date, binary, Extended JSON, array, struct, and missing/null rules; validate precision/overflow, recursive offsets, row-level rollback, and memory limits. Declare unsupported mappings explicitly.
• D05 — Safe predicate/LIKE pushdown and projection (W4–W7): classify translations as exact/superset/unsupported; preserve SQL three-valued logic, CAST, arrays, collation, and strict error semantics. Keep required residual predicates, implement conservative LIKE translation and column pruning, and gate behavior with Session variables. Include typed BSON Date time-range filtering for time-series collections. Standard-view SQL predicates remain local initially; do not override a view collation. Verify the existing SPI/applyFilter path cannot remove predicates incorrectly.
• D06 — Read policy, security, and resource control (W4–W7): implement Catalog defaults plus statement-pinned Session overrides for read preference/concern, sensitive configuration handling, TLS/credential versioning, pool isolation, batch/memory/time limits, and shared-source admission. Distinguish scanner-instance concurrency from query-wide and source-wide budgets.
  • Read-policy session overrides: implemented in [feature](catalog) Add MongoDB session read policies #70493 (pending merge), including mongodb_read_preference / mongodb_read_concern, inherit, SET / SET_VAR, validation, and statement-pinned scan policies. FE unit tests, both native SQL suites, and source-profiler checks on a primary/secondary replica set passed locally. The remaining D06 security/resource/admission work is still open.

M1 gate: basic reads of all three object kinds are mandatory. Validate view-output Schema, computed/projected/grouped fields, missing/non-unique _id, non-simple collation, and accounts that can query the view but cannot find the base collection. Validate time-series Schema without a validator, logical measurements, duplicate timestamps/metaField groups, BSON Date time filters, and cleanup. Result-multiset/error differential tests, permissions, pool isolation, and capability-rejection tests pass. Preview releases document single-cursor and consistency limits.

M2 — Parallel scans and a common snapshot

• D07a — Snapshot feasibility prototype (W1–W2, performed early): demonstrate a server-selected snapshot time T, its FE-to-BE representation, and the actual libmongoc command path that reads the same atClusterTime on replica sets and mongos. Majority reads alone do not satisfy this gate.
• D07b — Snapshot consistency implementation (W7–W9): pin the read contract across qualified Splits/objects in the same cluster; define history-expiration, identity/topology-change, cancellation, and retry behavior. Reject explicit common-snapshot queries containing standard views at planning time. Qualify time-series snapshot combinations against server/driver versions and concurrent-DDL restrictions separately. Basic view/time-series single-cursor reads remain required. Do not silently downgrade consistency.
• D08 — Bounded range/auto Split planning and execution (W8–W10): implement field/type-family selection, requested/admitted/actual Split counts, query/source budgets, bounded sampling, duplicate-boundary removal, and half-open ranges with unbounded ends. Preserve a disjoint complement for missing/null/array/other-type values when the scalar type family is not proven exhaustive. Freeze the complete plan before publishing any Split; expose decisions and fallback reasons in EXPLAIN/Profile. An already published plan must not fall back to a full single-Split rescan.

M2 gate: for qualified parallel-scan object kinds, every source document belongs to exactly one partition and the complete result multiset matches a single-Split read at the same snapshot. The required M2 parallel baseline covers ordinary collections; view/time-series basic reads continue through their declared single-cursor paths. Verify unsupported range/snapshot rejection, snapshot history failures, planning limits, scheduling, and source admission before enabling any additional mode.

Continuous validation and release qualification

• D09 — Regression and fault matrix (W1–W12, delivered with each implementation): provide controlled MongoDB services with environment-injected credentials; make ordinary-collection, standard-view, and time-series basic-read suites mandatory; cover view pipelines/output Schema/collation/least privileges and definition changes, time-series logical measurements/duplicate times/metaField/TTL/time filters, Schema sources/evolution, all mappings, SQL analytics, exact and superset pushdown, NULL/CAST/array counterexamples, malformed/deep/wide documents, duplicate/skewed split boundaries, complement partitions, authentication/TLS, topology changes, concurrent writes, timeout/cancellation, and resource cleanup. Add deterministic fixtures and reproducible differential checks.
• D10 — Performance, compatibility, and release evidence (W11–W12): establish repeatable single-cursor/pushdown/parallel baselines, source-side budgets, complete build/artifact qualification, supported driver/server/authentication combinations, EXPLAIN/Profile coverage, and user/operations documentation. Target at least 1,000 query/cancel/failure cycles and a 24-hour mixed-workload soak without continuously growing memory, threads, cursors, or connections. Run FE Checkstyle, clang-format 16 on affected BE files, and the relevant UT/regression checks.

Release gate: mandatory basic reads for all three object kinds, ordinary-collection M2, explicit unsupported-mode rejection, and all applicable correctness/resource/compatibility cases pass; archive reproducible CI, fault, and performance results. Week numbers are suggested relative development windows, not release commitments. The design reserves W13–W14 for blocking fixes and affected revalidation.

Dependency order

flowchart TD
    P["D00 dependencies + D02 query contract"] --> A["D01 Catalog / Schema: all three object kinds"]
    P --> B["D03 native Reader: logical namespaces"]
    P --> C["D04a basic DataTypeSerDe"]
    A --> M0["M0: single-cursor path"]
    B --> M0
    C --> M0
    P -. "validate in W1-W2" .-> S0["D07a snapshot prototype"]
    M0 --> T["D04b complete type semantics"]
    T --> F["D05 safe pushdown / projection"]
    M0 --> R["D06 read policy / resource admission"]
    F --> M1["M1: all three basic-read suites pass"]
    R --> M1
    M1 --> S["D07b fixed snapshot implementation"]
    S0 --> S
    M1 --> Q["D08 range / auto planner"]
    S --> M2["M2: qualified common snapshot + partitions + admission"]
    Q --> M2
    M2 --> RC["D10 release qualification"]

Independent branches may be developed in parallel. D01/D03/D05/D09 deliver view/time-series basic support by M1. Planner-only tests can start before snapshot implementation completes; enabling multi-Split reads requires both snapshot and partition/admission gates for the relevant object kind. D09 runs throughout all branches and milestones.

Follow-up capabilities, tracked separately after the initial release

• Advanced view/time-series optimizations only: separately prove safe view pushdown and timeField range planning, costs, and any supported time-series snapshot/DDL contract. Discovery, Schema, single-cursor reads, mappings, and basic filtering are already initial-release requirements. Source limitations on standard-view snapshot isolation remain explicit.
• Opt-in, bounded Schema inference with pinned versions; samples do not prove global type homogeneity.
• Whitelisted REGEXP, bounded runtime filters, LIMIT/TopN, and aggregation pushdown, each with a separate semantic contract, switch, differential suite, and benchmark.
• Source-side search interfaces; CDC, VARIANT document storage, and a large Schema registry require separate designs and lifecycles.

Tracking convention

Keep a work item unchecked until its implementation PRs and acceptance evidence are linked and verified. Link follow-up implementation PRs here and maintain the capability/support matrix as scope changes. The dependency preparation PR is not evidence that native MongoDB queries already work.