Skip to content

Optionaly return schema id together with decoded message - #3

Open
michaelr524 wants to merge 3 commits into
verisign:masterfrom
adqio:master
Open

Optionaly return schema id together with decoded message#3
michaelr524 wants to merge 3 commits into
verisign:masterfrom
adqio:master

Conversation

@michaelr524

Copy link
Copy Markdown

Useful for changing the record and encoding back using the same schema.

@ngoyal

ngoyal commented Feb 29, 2016

Copy link
Copy Markdown
Collaborator

LGTM. Deferring to @stevemil00 or @sean to accept.

@stevemil00

Copy link
Copy Markdown

@michaelr524 could you give a bit more detailed example of the use case you're describing? I'm having trouble thinking offhand of a case where that'd be all that useful, and I can't tell whether or not that just means I need more caffeine... :)

@michaelr524

Copy link
Copy Markdown
Author

@stevemil00 we have a spark-streaming job consuming a kafka topi, transforming the messages and writes them back to another kafka topic.
messages are encoded in avro of course.
since python-confluent-schemaregistry decodes avro messages to plain dictionaries there is no way to figure out the schema these messages were encoded in the source topic and hence no easy way to encode them back using the same schema.
by having the schema id returned from the decoder, i call this to encode the message back, along the way reusing the schema from the registry client cache:
encoded_message = serializer.encode_record_with_schema_id(schema_id, record)

@stevemil00

Copy link
Copy Markdown

LGTM, sorry for losing track of this. I don't actually have permissions to merge here so if @ngoyal or @sean could either give me permissions or push the button, that'd be great. :)

@miguno

miguno commented Mar 23, 2016

Copy link
Copy Markdown

@stevemil00 : I don't have admin permissions anymore either (I left the GH org) so I can't help you out in the meantime.

@miguno

miguno commented Mar 23, 2016

Copy link
Copy Markdown

That said, you might want to ensure there are updated unit tests for this new functionality (see e.g. https://github.com/verisign/python-confluent-schemaregistry/blob/master/test/test_message_serializer.py). I'd say this is important because the return type of decode_message changes (!) depending on its input parameters -- from a single value to a tuple. (Personally, I would have perhaps preferred a separate method like decode_message_with_schema rather than using an optional parameter that changes a function's return type.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants