Add schema to Schema Registry with a specific IdHandling bad messages using Kafka's Streams APIHow do I generate random integers within a specific range in Java?Confluent Platform: Schema Registry SubjectsSchema Registry persistence after rebootSchema issue with Kafka schema registryKafkaConnect HDFS Connector with SchemRegistryAdd New Schema/Subject to Schema Registry in Kafka Avro file formatConfluent 3.3 unable to connect to schema-registry locally with kafka-avro-console-producerError when use kafka-avro-console-consumer consume avro message sent by spring cloud stream kafkaUnknown magic byte with kafka-avro-console-consumerKafkaProducer is always picking localhost:8081 for schema registry in Java API
Why are on-board computers allowed to change controls without notifying the pilots?
My boss asked me to take a one-day class, then signs it up as a day off
Reply ‘no position’ while the job posting is still there (‘HiWi’ position in Germany)
Is there any significance to the Valyrian Stone vault door of Qarth?
What if somebody invests in my application?
Giant Toughroad SLR 2 for 200 miles in two days, will it make it?
What is the opposite of 'gravitas'?
How do ultrasonic sensors differentiate between transmitted and received signals?
Pronouncing Homer as in modern Greek
What does the "3am" section means in manpages?
Hostile work environment after whistle-blowing on coworker and our boss. What do I do?
Why isn't KTEX's runway designation 10/28 instead of 9/27?
Can I create an upright 7ft x 5ft wall with Minor Illusion?
What (else) happened July 1st 1858 in London?
Bob has never been a M before
Have I saved too much for retirement so far?
Female=gender counterpart?
Can I use my Chinese passport to enter China after I acquired another citizenship?
How to deal with or prevent idle in the test team?
I'm in charge of equipment buying but no one's ever happy with what I choose. How to fix this?
How will losing mobility of one hand affect my career as a programmer?
What to do when my ideas aren't chosen, when I strongly disagree with the chosen solution?
Installing PowerShell on 32-bit Kali OS fails
Is there a problem with hiding "forgot password" until it's needed?
Add schema to Schema Registry with a specific Id
Handling bad messages using Kafka's Streams APIHow do I generate random integers within a specific range in Java?Confluent Platform: Schema Registry SubjectsSchema Registry persistence after rebootSchema issue with Kafka schema registryKafkaConnect HDFS Connector with SchemRegistryAdd New Schema/Subject to Schema Registry in Kafka Avro file formatConfluent 3.3 unable to connect to schema-registry locally with kafka-avro-console-producerError when use kafka-avro-console-consumer consume avro message sent by spring cloud stream kafkaUnknown magic byte with kafka-avro-console-consumerKafkaProducer is always picking localhost:8081 for schema registry in Java API
We have been using Confluent Schema Registry with KafkaStreams for over a year now and everything has been working well; until yesterday.
In an UAT environment we seem to have had a Schema subject deleted and one of our applications started failing over with the message
[ERROR] LogAndFailExceptionHandler - Exception caught during
Deserialization, taskId: 0_13, topic: TOPIC_NAME, partition: 13,
offset: 0 org.apache.kafka.common.errors.SerializationException: Error
retrieving Avro schema for id 1531
I checked the Schema Registry and noticed the subject was missing and queries the id 1531 that is listed in the error with curl such as:
curl -X GET http://SchemaRegistryHost:8081/schemas/ids/1531
And got back:
"error_code":40403,"message":"Schema not found"
I naively just tried to register the schema again without thinking about it and it worked but the id the schema was registered with was not the same as the previous 1531 ID.
I need the schema registered to the ID 1531 since the existing messages in the topic already contain that Id 1531 in the magic byte.
I checked the API docs at https://docs.confluent.io/current/schema-registry/docs/develop/api.html but did not see anything for setting a given Id for a schema.
Is there anyway to force a schema to a specific Id with schema registry?
I am aware of some backup solutions but I am looking for a fix now that will hopefully prevent loss of data or extraordinary measures to fix the topic data.
java apache-kafka confluent-schema-registry
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
asked Mar 21 at 14:38
DVSDVS
318212
add a comment |
We have been using Confluent Schema Registry with KafkaStreams for over a year now and everything has been working well; until yesterday.
In an UAT environment we seem to have had a Schema subject deleted and one of our applications started failing over with the message
[ERROR] LogAndFailExceptionHandler - Exception caught during
Deserialization, taskId: 0_13, topic: TOPIC_NAME, partition: 13,
offset: 0 org.apache.kafka.common.errors.SerializationException: Error
retrieving Avro schema for id 1531
I checked the Schema Registry and noticed the subject was missing and queries the id 1531 that is listed in the error with curl such as:
curl -X GET http://SchemaRegistryHost:8081/schemas/ids/1531
And got back:
"error_code":40403,"message":"Schema not found"
I naively just tried to register the schema again without thinking about it and it worked but the id the schema was registered with was not the same as the previous 1531 ID.
I need the schema registered to the ID 1531 since the existing messages in the topic already contain that Id 1531 in the magic byte.
I checked the API docs at https://docs.confluent.io/current/schema-registry/docs/develop/api.html but did not see anything for setting a given Id for a schema.
Is there anyway to force a schema to a specific Id with schema registry?
I am aware of some backup solutions but I am looking for a fix now that will hopefully prevent loss of data or extraordinary measures to fix the topic data.
java apache-kafka confluent-schema-registry
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
asked Mar 21 at 14:38
DVSDVS
318212
add a comment |
We have been using Confluent Schema Registry with KafkaStreams for over a year now and everything has been working well; until yesterday.
In an UAT environment we seem to have had a Schema subject deleted and one of our applications started failing over with the message
[ERROR] LogAndFailExceptionHandler - Exception caught during
Deserialization, taskId: 0_13, topic: TOPIC_NAME, partition: 13,
offset: 0 org.apache.kafka.common.errors.SerializationException: Error
retrieving Avro schema for id 1531
I checked the Schema Registry and noticed the subject was missing and queries the id 1531 that is listed in the error with curl such as:
curl -X GET http://SchemaRegistryHost:8081/schemas/ids/1531
And got back:
"error_code":40403,"message":"Schema not found"
I naively just tried to register the schema again without thinking about it and it worked but the id the schema was registered with was not the same as the previous 1531 ID.
I need the schema registered to the ID 1531 since the existing messages in the topic already contain that Id 1531 in the magic byte.
I checked the API docs at https://docs.confluent.io/current/schema-registry/docs/develop/api.html but did not see anything for setting a given Id for a schema.
Is there anyway to force a schema to a specific Id with schema registry?
I am aware of some backup solutions but I am looking for a fix now that will hopefully prevent loss of data or extraordinary measures to fix the topic data.
java apache-kafka confluent-schema-registry
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
asked Mar 21 at 14:38
DVSDVS
318212
We have been using Confluent Schema Registry with KafkaStreams for over a year now and everything has been working well; until yesterday.
In an UAT environment we seem to have had a Schema subject deleted and one of our applications started failing over with the message
[ERROR] LogAndFailExceptionHandler - Exception caught during
Deserialization, taskId: 0_13, topic: TOPIC_NAME, partition: 13,
offset: 0 org.apache.kafka.common.errors.SerializationException: Error
retrieving Avro schema for id 1531
I checked the Schema Registry and noticed the subject was missing and queries the id 1531 that is listed in the error with curl such as:
curl -X GET http://SchemaRegistryHost:8081/schemas/ids/1531
And got back:
"error_code":40403,"message":"Schema not found"
I naively just tried to register the schema again without thinking about it and it worked but the id the schema was registered with was not the same as the previous 1531 ID.
I need the schema registered to the ID 1531 since the existing messages in the topic already contain that Id 1531 in the magic byte.
I checked the API docs at https://docs.confluent.io/current/schema-registry/docs/develop/api.html but did not see anything for setting a given Id for a schema.
Is there anyway to force a schema to a specific Id with schema registry?
I am aware of some backup solutions but I am looking for a fix now that will hopefully prevent loss of data or extraordinary measures to fix the topic data.
java apache-kafka confluent-schema-registry
java apache-kafka confluent-schema-registry
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
asked Mar 21 at 14:38
DVSDVS
318212
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
asked Mar 21 at 14:38
DVSDVS
318212
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
edited Mar 22 at 7:04
Matthias J. Sax
31.3k45582
31.3k45582
asked Mar 21 at 14:38
DVSDVS
318212
asked Mar 21 at 14:38
DVSDVS
318212
asked Mar 21 at 14:38
DVSDVS
318212
318212
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Is there anyway to force a schema to a specific Id with schema registry?
There is not.
The ID of 1531 isn't actually "gone", by the way, it just is marked as deleted in the registry (consume the _schemas topic to see it).
There really is no way around the error that I know of while you use the KafkaAvroDeserializer. You would have to use the ByteArrayDeserializer, then "fix" or "lookup" the correct ID using the Schema Registry client, and then deserialize the rest of the message.
The other option would be to reset your consumer group such that you skip these messages entirely, or setup exception handling. Handling bad messages using Kafka's Streams API
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
The good news is that this is only a UAT environment so if we "skip" data it isn't the end of the world but it seems like had this happened in a production environment where there is actual financial cost with skipping messages this would not be an acceptable solution. Also seems getting production schema registry backed up and instituting better controls over deleting schema subjects may be very critical... though it seems like confluent should consider extending the API allow for such a use-case in these type of recovery situations.
– DVS
Mar 22 at 13:59
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55282944%2fadd-schema-to-schema-registry-with-a-specific-id%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is there anyway to force a schema to a specific Id with schema registry?
There is not.
The ID of 1531 isn't actually "gone", by the way, it just is marked as deleted in the registry (consume the _schemas topic to see it).
There really is no way around the error that I know of while you use the KafkaAvroDeserializer. You would have to use the ByteArrayDeserializer, then "fix" or "lookup" the correct ID using the Schema Registry client, and then deserialize the rest of the message.
The other option would be to reset your consumer group such that you skip these messages entirely, or setup exception handling. Handling bad messages using Kafka's Streams API
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
The good news is that this is only a UAT environment so if we "skip" data it isn't the end of the world but it seems like had this happened in a production environment where there is actual financial cost with skipping messages this would not be an acceptable solution. Also seems getting production schema registry backed up and instituting better controls over deleting schema subjects may be very critical... though it seems like confluent should consider extending the API allow for such a use-case in these type of recovery situations.
– DVS
Mar 22 at 13:59
add a comment |
Is there anyway to force a schema to a specific Id with schema registry?
There is not.
The ID of 1531 isn't actually "gone", by the way, it just is marked as deleted in the registry (consume the _schemas topic to see it).
There really is no way around the error that I know of while you use the KafkaAvroDeserializer. You would have to use the ByteArrayDeserializer, then "fix" or "lookup" the correct ID using the Schema Registry client, and then deserialize the rest of the message.
The other option would be to reset your consumer group such that you skip these messages entirely, or setup exception handling. Handling bad messages using Kafka's Streams API
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
The good news is that this is only a UAT environment so if we "skip" data it isn't the end of the world but it seems like had this happened in a production environment where there is actual financial cost with skipping messages this would not be an acceptable solution. Also seems getting production schema registry backed up and instituting better controls over deleting schema subjects may be very critical... though it seems like confluent should consider extending the API allow for such a use-case in these type of recovery situations.
– DVS
Mar 22 at 13:59
add a comment |
Is there anyway to force a schema to a specific Id with schema registry?
There is not.
The ID of 1531 isn't actually "gone", by the way, it just is marked as deleted in the registry (consume the _schemas topic to see it).
There really is no way around the error that I know of while you use the KafkaAvroDeserializer. You would have to use the ByteArrayDeserializer, then "fix" or "lookup" the correct ID using the Schema Registry client, and then deserialize the rest of the message.
The other option would be to reset your consumer group such that you skip these messages entirely, or setup exception handling. Handling bad messages using Kafka's Streams API
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
Is there anyway to force a schema to a specific Id with schema registry?
There is not.
The ID of 1531 isn't actually "gone", by the way, it just is marked as deleted in the registry (consume the _schemas topic to see it).
There really is no way around the error that I know of while you use the KafkaAvroDeserializer. You would have to use the ByteArrayDeserializer, then "fix" or "lookup" the correct ID using the Schema Registry client, and then deserialize the rest of the message.
The other option would be to reset your consumer group such that you skip these messages entirely, or setup exception handling. Handling bad messages using Kafka's Streams API
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
answered Mar 21 at 18:42
cricket_007cricket_007
83.5k1146116
83.5k1146116
The good news is that this is only a UAT environment so if we "skip" data it isn't the end of the world but it seems like had this happened in a production environment where there is actual financial cost with skipping messages this would not be an acceptable solution. Also seems getting production schema registry backed up and instituting better controls over deleting schema subjects may be very critical... though it seems like confluent should consider extending the API allow for such a use-case in these type of recovery situations.
– DVS
Mar 22 at 13:59
add a comment |
The good news is that this is only a UAT environment so if we "skip" data it isn't the end of the world but it seems like had this happened in a production environment where there is actual financial cost with skipping messages this would not be an acceptable solution. Also seems getting production schema registry backed up and instituting better controls over deleting schema subjects may be very critical... though it seems like confluent should consider extending the API allow for such a use-case in these type of recovery situations.
– DVS
Mar 22 at 13:59
The good news is that this is only a UAT environment so if we "skip" data it isn't the end of the world but it seems like had this happened in a production environment where there is actual financial cost with skipping messages this would not be an acceptable solution. Also seems getting production schema registry backed up and instituting better controls over deleting schema subjects may be very critical... though it seems like confluent should consider extending the API allow for such a use-case in these type of recovery situations.
– DVS
Mar 22 at 13:59
The good news is that this is only a UAT environment so if we "skip" data it isn't the end of the world but it seems like had this happened in a production environment where there is actual financial cost with skipping messages this would not be an acceptable solution. Also seems getting production schema registry backed up and instituting better controls over deleting schema subjects may be very critical... though it seems like confluent should consider extending the API allow for such a use-case in these type of recovery situations.
– DVS
Mar 22 at 13:59
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55282944%2fadd-schema-to-schema-registry-with-a-specific-id%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
