support filtering out deletion operation types for mongo by jgao54 · Pull Request #4585 · PeerDB-io/peerdb

jgao54 · 2026-07-15T01:55:53Z

Allow filtering out operations by type. This is already possible today with Postgres publications. Here we extend this feature to MongoDB by filtering out operations in the pipeline definition. This allows data to be filtered out on the server-side.

Adding a UI component in Edit Mirror page that supports updating the setting for a mirror (vs. the settings tab, that would update the setting for all pipes). If both are set, pipe-level setting will take precedence. Here we introduce a source-type-specific allowlist so the settings override shows up for applicable pipes (in this case, only Mongo pipe will display this option). Note that the CDCFlowConfigUpdate field in FlowStateChangeRequest already supports updatedEnv today, so this PR just surfaces it to the UI.

Test: added uinit/e2e tests; also verified locally e2e that pipeline filtering works as expected and can be modified dynamically on pause/resume.

Fixes: DBI-922

jgao54 · 2026-07-15T06:47:27Z

+	// Unlike Postgres, MongoDB doesn't need a dedicated replication connection:
+	// change streams are cursors served by the connector's pooled client.
+	// Since SetupReplConn is called once per SyncFlow activity, resolving
+	// dynamic config here avoids per-batch catalog reads.


Also evaluated populating this in the mongo connector itself similar to Postgres, but this is a more logical place for it despite the SetupReplConn naming is a bit misleading. Not all mongo connectors instantiated need to worry about operation type exclusion, in fact only cdc does. So populating excludedOps here feels like the more natural placement. Although I can be persuaded otherwise.

I think we have a few other cdc-related settings that are being fetched every batch today (when we expect them to be fetched every SyncFlow and only update on pause/resume), so they behave more like APPLY_MODE_IMMEDIATE even though they are labeled APPLY_MODE_AFTER_RESUME, resulting in a lot of db calls that could be avoided. So one of the follow-up would be to identify these dynconf and move them here as well.

claude · 2026-07-15T06:51:40Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

morsapaes · 2026-07-15T08:48:52Z

+// which dynLookup checks before the service-level dynamic_settings catalog value.
+const pipeLevelSettings: { name: string; sourceTypes: DBType[] }[] = [
+  {
+    name: 'PEERDB_MONGODB_EXCLUDED_OPERATION_TYPES',


Can we expose the setting with a more user-friendly name in the UI vs. the raw setting name? In a similar direction, "Setting Overrides" bleeds implementation details; if we expose it as a configurable setting, it should be implied it's a safe override. Maybe "Replication Settings" would be more appropriate and future-proof, with a helper text like "Changes apply to this mirror only and take precedence over global defaults"?

ah yeah, that's a good idea for the pipe-level setting override.

one downside is that this will make it less obvious it's referring to the same global setting here, which we already expose as raw envs in the Settings page, and it will also apply to other global envs that we want to enable on the pipe-level in the future.

We already have a system for overriding settings on the mirror level, some of our OSS customers are using that for the multi-tenancy setup. The UI is even clunkier (text box accepting raw JSON) but there is a precedent and if we're introducing support in editing, those probably should be unified (at some UX level).

peerdb/ui/app/mirrors/create/helpers/cdc.ts

Lines 227 to 238 in 58babbd

{

label: 'Settings override',

stateHandler: (value, setter) =>

setter((curr: CDCConfig): CDCConfig => ({

...curr,

envString: value as string,

})),

type: 'textarea',

default: '',

tips: '{"string":"string"} JSON mapping to override global settings for mirror',

advanced: AdvancedSettingType.ALL,

},

morsapaes · 2026-07-15T08:51:15Z

 		TargetForSetting: protos.DynconfTarget_ALL,
 	},
+	{
+		Name: "PEERDB_MONGODB_EXCLUDED_OPERATION_TYPES",


Since the set of options is bound and known, should this be a multi-select field instead to reduce the risk of typos? Not sure if we have building blocks for this in the UI components, so a comma-separated list is fine to start; just wanted to flag.

For global settings tab, this isn't super trivial to change; but for the pipe-level configs we can build something more user-friendly and translate it to env var on our end

pfcoperez

LGTM, conditioned on the log message change and on the confirmation that dups are inconsequential.

This reverts commit 3b4c4f5.

jgao54 · 2026-07-15T21:06:42Z

Removed the UI component for now, will address as a follow-up. In the mean time the env can be updated via flow-api:

// pause the pipe
curl -X POST localhost:8113/v1/mirrors/state_change \
  -H "Content-Type: application/json" \
  -d '{"flow_job_name": "mongo_filter_test", "requested_flow_state": "STATUS_PAUSED"}'

// edit env + resume the pipe
curl -X POST localhost:8113/v1/mirrors/state_change \
  -H "Content-Type: application/json" \
  -d '{
    "flow_job_name": "mongo_filter_test",
    "requested_flow_state": "STATUS_RUNNING",
    "flow_config_update": {
      "cdc_flow_config_update": {
        "updated_env": {"PEERDB_MONGODB_EXCLUDED_OPERATION_TYPES": "delete"}
      }
    }
  }'

codecov · 2026-07-15T21:12:35Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
3064	2	3062	377

View the top 2 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg

Stack Traces | 0.01s run time

=== RUN   TestApiPg
=== PAUSE TestApiPg
=== CONT  TestApiPg
--- FAIL: TestApiPg (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew

Stack Traces | 4.15s run time

=== RUN   TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
=== PAUSE TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
=== CONT  TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
2026/07/15 20:59:15 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/15 20:59:15 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    api_test.go:3575: WaitFor wait for flow to be running 2026-07-15 20:59:19.771564078 +0000 UTC m=+252.160255773
    api_test.go:3586: 
        	Error Trace:	.../flow/e2e/api_test.go:3586
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	"1" is not greater than "1"
        	Test:       	TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew
        	Messages:   	Should have multiple executions (continue-as-new happened)
    api_test.go:51: begin tearing down postgres schema api_neqr3iuz
--- FAIL: TestApiPg/TestCreateCDCFlowAttachIdempotentAfterContinueAsNew (4.15s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

jgao54 force-pushed the support-filter-ops-types branch 3 times, most recently from 4c561d9 to 9713f30 Compare July 15, 2026 06:31

support filtering out deletion operation types for mongo

4204ede

jgao54 force-pushed the support-filter-ops-types branch from 9713f30 to 4204ede Compare July 15, 2026 06:34

jgao54 commented Jul 15, 2026

View reviewed changes

jgao54 marked this pull request as ready for review July 15, 2026 06:47

jgao54 requested a review from a team as a code owner July 15, 2026 06:47

masterashu approved these changes Jul 15, 2026

View reviewed changes

UI to allow pipe-level env editing

3b4c4f5

dtunikov approved these changes Jul 15, 2026

View reviewed changes

pfcoperez reviewed Jul 15, 2026

View reviewed changes

Comment thread flow/connectors/mongo/cdc.go Outdated

morsapaes reviewed Jul 15, 2026

View reviewed changes

pfcoperez reviewed Jul 15, 2026

View reviewed changes

Comment thread flow/connectors/mongo/cdc.go Outdated

morsapaes reviewed Jul 15, 2026

View reviewed changes

pfcoperez reviewed Jul 15, 2026

View reviewed changes

Comment thread flow/connectors/mongo/cdc.go

pfcoperez reviewed Jul 15, 2026

View reviewed changes

Comment thread flow/connectors/mongo/cdc.go Outdated

pfcoperez approved these changes Jul 15, 2026

View reviewed changes

jgao54 added 2 commits July 15, 2026 10:41

review

edf96d3

Revert "UI to allow pipe-level env editing"

05d9857

This reverts commit 3b4c4f5.

jgao54 merged commit a1fa530 into main Jul 15, 2026
20 of 21 checks passed

jgao54 deleted the support-filter-ops-types branch July 15, 2026 21:41

	{
	label: 'Settings override',
	stateHandler: (value, setter) =>
	setter((curr: CDCConfig): CDCConfig => ({
	...curr,
	envString: value as string,
	})),
	type: 'textarea',
	default: '',
	tips: '{"string":"string"} JSON mapping to override global settings for mirror',
	advanced: AdvancedSettingType.ALL,
	},

Uh oh!

Conversation

jgao54 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgao54 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Jul 15, 2026

Code review

Uh oh!

Uh oh!

morsapaes Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilidemi Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

morsapaes Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pfcoperez left a comment

Choose a reason for hiding this comment

Uh oh!

jgao54 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 15, 2026

❌ 2 Tests Failed:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jgao54 commented Jul 15, 2026 •

edited

Loading

jgao54 Jul 15, 2026 •

edited

Loading

jgao54 commented Jul 15, 2026 •

edited

Loading