codegen(meos): generate tier-aware MEOS facade for the full JMEOS 1.4 surface (Kafka mirror of MobilityFlink #5; stacks on #2) by estebanzimanyi · Pull Request #3 · MobilityDB/MobilityKafka

estebanzimanyi · 2026-05-21T10:15:34Z

Add a generated, tier-aware Java facade over the full MEOS public API surface — Kafka Streams mirror of MobilityFlink #5.

What is generated

Identical shape to MobilityFlink #5 — same generators, same tier classification, same catalog source. Differs only in:

Package: org.mobilitydb.kafka.meos (vs org.mobilitydb.flink.meos)
Module: kafka-streams-app/ (vs flink-processor/)

Layer	Java class shape	Files	Methods
OO-classified	`MeosOps<Class>` — one per MEOS object-model class	50	751
Free fns	`MeosOpsFree<Header>` — one per public MEOS header for fns not assigned to any OO class	6	1,346
Shared runtime	`MeosOpsRuntime` (singleton `MEOS_AVAILABLE`)	1	—
Total		57	2,097 (77.7% of JMEOS PR #19's 2,699-method surface)

Tier breakdown: 804 stateless · 797 bounded-state · 161 windowed · 140 cross-stream · 195 io-meta.

Each emitted method forwards verbatim to functions.GeneratedFunctions.<name>(...) after probing MeosOpsRuntime.MEOS_AVAILABLE. Method Javadocs carry tier markers — see MobilityFlink #5 for the full tier vocabulary table.

Kafka Streams wiring per tier

Tier	Kafka Streams pattern
`stateless` / `bounded-state`	`mapValues(...)` / `filter(...)` — MEOS handle is per-event state
`windowed`	`groupByKey().windowedBy(TimeWindows...).aggregate(...)`
`cross-stream`	`KStream-KStream` join with windowed state store
`io-meta`	Serde / format helpers
`sequence-only` (14 fns, deferred)	Excluded — inherently non-streamable

Coexistence with `berlinmod.MEOSBridge`

Same coexistence pattern as MobilityFlink #5: MEOSBridge keeps the per-BerlinMOD-query intent; MeosOps* exposes the raw MEOS surface tier-by-tier. Both share the same MEOS_AVAILABLE discipline (via MeosOpsRuntime) and functions.GeneratedFunctions delegation.

Stacking

This PR stacks on feat/jmeos-bridge-swap. Additive-only: 57 new Java files + 5 files under tools/codegen/. No existing file is touched.

Base-branch compile state

Same situation as MobilityFlink — the recent bundled-jar refresh (c28fd83, JMEOS PR #19's jmeos-core jar) brought in the 2,699-method functions.GeneratedFunctions surface but did not include JMEOS PR #18's utils.spatial.* wrappers that the parallel session's berlinmod.MEOSBridge imports. Recipe to refresh with the union jar is in MobilityFlink PR #5's body. This PR's own diff is green in isolation against the refreshed jar.

Regeneration

See kafka-streams-app/tools/codegen/README.md (and the parallel flink-processor/tools/codegen/README.md on MobilityFlink #5 — same recipe).

estebanzimanyi · 2026-05-21T11:03:04Z

Coordination confirmation: rebased onto the post-union-jar refresh (6676bbb on Flink / fa70867 on Kafka — the JMEOS PR #19 + PR #18 union jar with utils.spatial.{Haversine,PointToSegment}).

Local verification:

$ mvn -q -DskipTests compile
# Flink: 123 .class files total, of which 57 MeosOps*
# Kafka:  94 .class files total, of which 57 MeosOps*

Full module now compiles green — the codegen wedge sits on top of MEOSBridge.java's utils.spatial.* imports without any inherited compile-red. mergeStateStatus = CLEAN. Ready for review/merge whenever.

Coordination item resolved. Thanks for the union-jar refresh.

… surface Kafka mirror of MobilityFlink: regenerated from JMEOS PR #19's GeneratedFunctions (accumulate/parity-1.4 IDL) — renamed trgeometry_* surface + type-catalog/ transform/talpha (60 MeosOps* classes, 2,166 methods); bundled JMEOS.jar GeneratedFunctions refreshed. Compiles clean; legacy functions.functions surface untouched.

estebanzimanyi · 2026-06-10T15:09:11Z

Consolidated into #13 (consolidate/kafka-streaming-implementation).

…eams (27/27) All 27 cells of the BerlinMOD-9 × 3-form parity matrix on MobilityKafka, matching MobilityFlink MobilityDB#3's coverage on the Kafka-Streams runtime. Continuous form (9): per-event emission via record-by-record dispatch. Windowed form (9): STREAM_TIME punctuator at WINDOW_SIZE_MILLIS (10_000 ms) emitting closed windows. Snapshot form (9): STREAM_TIME punctuator at SNAPSHOT_TICK_MILLIS (5_000 ms) emitting per-tick state. State patterns: - Stateless filter (Q2-c, Q3-c, Q8-c) - Single keyed flag (Q1-c, Q1-s, Q4-c) - Single-key state (Q2-s, Q9-s) - Per-vehicle accumulator (Q6-c, Q6-s) - Per-(vehicle, POI) keyed state (Q7-c, Q7-s) - Cross-vehicle shared via selectKey(0) (Q5-c, Q3-s, Q5-s, Q7-s, Q8-s) - Paired shared via selectKey(0) (Q9-c, Q4-s, Q9-s) - winStart-keyed comma-separated vehicleId set (Q1-w, Q3-w, Q8-w) - winStart-keyed encoded last-known per vehicle (Q2-w, Q5-w, Q9-w) - winStart-keyed per-vehicle accumulator (Q6-w) - winStart-keyed per-vehicle entries log (Q4-w, Q7-w) Output counts (TopologyTestDriver, 21-event sorted-by-event-time corpus + 2 sentinel records at t=15001 and t=20001): Q | continuous | windowed | snapshot ---|------------|----------|--------- Q1 | 3 | 2 | 13 Q2 | 7 | 2 | 4 Q3 | 21 | 2 | 9 Q4 | 1 | 2 | 5 Q5 | 19 | 2 | 4 Q6 | 21 | 6 | 13 Q7 | 3 | 6 | 13 Q8 | 21 | 2 | 9 Q9 | 13 | 2 | 4 All 9 windowed cells emit 2 lines per Q (Q1/Q2/Q3/Q4/Q5/Q8/Q9) or 6 lines per Q (Q6/Q7 — one per vehicle × 2 windows for per-vehicle outputs). Continuous and snapshot count differences vs MobilityFlink reflect Kafka Streams' STREAM_TIME punctuator semantics: fires on stream-time advance with state-at-fire-moment, multi-interval jumps coalesced; vs Flink's bounded source flushing all keyed timers with final state at +infty. Dual-sentinel pattern in LocalTest steps the punctuator through the desired tick boundaries. Bump-isolation: zero JMEOS calls, zero MEOS-C calls, zero PyMEOS dependency. Pure Kafka Streams + Jackson + Java. Spatial predicates use Haversine / SegmentDistance / point-in-box, marked TODO(meos) for JMEOS-bridge migration.

estebanzimanyi · 2026-06-11T17:25:47Z

Superseded by the Path-B consolidation: the former 12-deep stack is collapsed into three reviewable topical PRs — scaffold #1 → MEOS integration #14 → benchmark #15 — each one clean squashed commit with the generated-facade bulk, dead family-flag profiles, and invented synthetic corpus removed. Closing as folded into #14/#15.

…eams (27/27) All 27 cells of the BerlinMOD-9 × 3-form parity matrix on MobilityKafka, matching MobilityFlink MobilityDB#3's coverage on the Kafka-Streams runtime. Continuous form (9): per-event emission via record-by-record dispatch. Windowed form (9): STREAM_TIME punctuator at WINDOW_SIZE_MILLIS (10_000 ms) emitting closed windows. Snapshot form (9): STREAM_TIME punctuator at SNAPSHOT_TICK_MILLIS (5_000 ms) emitting per-tick state. State patterns: - Stateless filter (Q2-c, Q3-c, Q8-c) - Single keyed flag (Q1-c, Q1-s, Q4-c) - Single-key state (Q2-s, Q9-s) - Per-vehicle accumulator (Q6-c, Q6-s) - Per-(vehicle, POI) keyed state (Q7-c, Q7-s) - Cross-vehicle shared via selectKey(0) (Q5-c, Q3-s, Q5-s, Q7-s, Q8-s) - Paired shared via selectKey(0) (Q9-c, Q4-s, Q9-s) - winStart-keyed comma-separated vehicleId set (Q1-w, Q3-w, Q8-w) - winStart-keyed encoded last-known per vehicle (Q2-w, Q5-w, Q9-w) - winStart-keyed per-vehicle accumulator (Q6-w) - winStart-keyed per-vehicle entries log (Q4-w, Q7-w) Output counts (TopologyTestDriver, 21-event sorted-by-event-time corpus + 2 sentinel records at t=15001 and t=20001): Q | continuous | windowed | snapshot ---|------------|----------|--------- Q1 | 3 | 2 | 13 Q2 | 7 | 2 | 4 Q3 | 21 | 2 | 9 Q4 | 1 | 2 | 5 Q5 | 19 | 2 | 4 Q6 | 21 | 6 | 13 Q7 | 3 | 6 | 13 Q8 | 21 | 2 | 9 Q9 | 13 | 2 | 4 All 9 windowed cells emit 2 lines per Q (Q1/Q2/Q3/Q4/Q5/Q8/Q9) or 6 lines per Q (Q6/Q7 — one per vehicle × 2 windows for per-vehicle outputs). Continuous and snapshot count differences vs MobilityFlink reflect Kafka Streams' STREAM_TIME punctuator semantics: fires on stream-time advance with state-at-fire-moment, multi-interval jumps coalesced; vs Flink's bounded source flushing all keyed timers with final state at +infty. Dual-sentinel pattern in LocalTest steps the punctuator through the desired tick boundaries. Bump-isolation: zero JMEOS calls, zero MEOS-C calls, zero PyMEOS dependency. Pure Kafka Streams + Jackson + Java. Spatial predicates use Haversine / SegmentDistance / point-in-box, marked TODO(meos) for JMEOS-bridge migration.

estebanzimanyi force-pushed the codegen/kafka-meos-ops branch from a1d7fbf to 9ac326f Compare May 21, 2026 11:02

estebanzimanyi mentioned this pull request May 21, 2026

feat(wirings): Kafka Streams DSL tier wirings + capstone demo (stacks on #3; Kafka mirror of Flink #6→#10) #4

Closed

estebanzimanyi force-pushed the codegen/kafka-meos-ops branch from 9ac326f to e1027c3 Compare May 23, 2026 08:58

estebanzimanyi mentioned this pull request May 21, 2026

feat(berlinmod): full BerlinMOD-9 × 3-form parity matrix on Kafka Streams (27/27 cells) #1

Open

estebanzimanyi closed this Jun 10, 2026

estebanzimanyi reopened this Jun 10, 2026

estebanzimanyi changed the base branch from main to feat/jmeos-bridge-swap June 11, 2026 16:44

estebanzimanyi closed this Jun 11, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

codegen(meos): generate tier-aware MEOS facade for the full JMEOS 1.4 surface (Kafka mirror of MobilityFlink #5; stacks on #2)#3

codegen(meos): generate tier-aware MEOS facade for the full JMEOS 1.4 surface (Kafka mirror of MobilityFlink #5; stacks on #2)#3
estebanzimanyi wants to merge 1 commit into
MobilityDB:feat/jmeos-bridge-swapfrom
estebanzimanyi:codegen/kafka-meos-ops

estebanzimanyi commented May 21, 2026

Uh oh!

estebanzimanyi commented May 21, 2026

Uh oh!

estebanzimanyi commented Jun 10, 2026

Uh oh!

estebanzimanyi commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

estebanzimanyi commented May 21, 2026

What is generated

Kafka Streams wiring per tier

Coexistence with berlinmod.MEOSBridge

Stacking

Base-branch compile state

Regeneration

Uh oh!

estebanzimanyi commented May 21, 2026

Uh oh!

estebanzimanyi commented Jun 10, 2026

Uh oh!

estebanzimanyi commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Coexistence with `berlinmod.MEOSBridge`