Add MODE 3 implementation (ingestion with Kafka consumer)#25
Merged
Conversation
gmunozfe
requested changes
May 27, 2026
gmunozfe
left a comment
gmunozfe
left a comment
Contributor
There was a problem hiding this comment.
Two things to fix:
- Reject blank workflow/task identifiers. The current workflow path defaults missing data.name to "", and processors only reject null IDs, so invalid events can create an empty workflow ID.
- Align status mapping with the existing Data Index model. task.started.v1 currently maps to STARTED, while other modes use RUNNING.
mcruzdev
force-pushed
the
issue-23
branch
4 times, most recently
from
05b7844 to
a92af73
Compare
|
|
||
| import java.time.OffsetDateTime; | ||
|
|
||
| public class LifecycleEvent { |
Contributor
There was a problem hiding this comment.
Don't we have this already in the SDK?
Contributor
Author
There was a problem hiding this comment.
Yes, we have a lot of classes (WorkflowCompletedCEData, WorkflowStartedCEData, TaskCancelledCEData) that represent each kind of lifecycle event, but this class aims to merge everything in a centralized data transfer object to be parsed.
Contributor
There was a problem hiding this comment.
Why cannot deserialize to the proper SDK POJO using the cloud event type ?
There is a clear and predicatble mapping between each type and each pojo as defined in the SDK, there is not need to create a union class holding all the structs.
Also, now that the SDK is a hierarchy after the later changes you can add common code to handle the different types generically (WorkflowCEEvent and TaskCEEvent) and just an extension to deal with different date fields names
gmunozfe
reviewed
May 31, 2026
Signed-off-by: Matheus Cruz <matheuscruz.dev@gmail.com>
fjtirado
reviewed
Jun 3, 2026
Contributor
There was a problem hiding this comment.
I would use InputStreamReader and BufferedReader#readAllAsString() here.
fjtirado
reviewed
Jun 3, 2026
Signed-off-by: Matheus Cruz <matheuscruz.dev@gmail.com>
fjtirado
approved these changes
Jun 5, 2026
Contributor
ricardozanini
approved these changes
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces production-ready support and documentation for MODE 3 (Kafka-based ingestion) in the Data Index project, alongside improvements to the documentation build process. The main changes add comprehensive guidance, architecture, and code references for Kafka ingestion, clarify when to use each mode, and update the documentation system to support both Maven and npm workflows.
MODE 3 (Kafka) Support and Documentation:
data-index/data-index-docs/modules/ROOT/pages/architecture/kafka-mode.adoc, covering event pipeline, components, persistence, idempotency, out-of-order handling, error processing, and a comparison with other modes.CLAUDE.mdto reflect MODE 3 as production ready, describe its architecture, clarify when to use it, and list new key components, code structure, and integration tests for Kafka ingestion. [1] [2] [3] [4] [5] [6] [7] [8]Documentation Navigation and Structure:
nav.adoc). [1] [2]nodemon. [1] [2] [3] [4]These changes collectively enable and document the Kafka-based ingestion (MODE 3) as a first-class, production-ready option, and modernize the documentation workflow for contributors.