Cannot start kafka connect distributed using windows bat file Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!Java Picasa API Directory listing not workingException in thread “main” java.lang.NoClassDefFoundError: org/apache/logging/log4j/message/MessageJDBC MySQL ConnectorGetting error in executing OWL APIHow to connect to mysql database using java?Unable to Setup CoAP Server using Californium Java LibraryHow to connect my Java applicationan to an Oracle databasespark kafka streaming Error - " java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtilsjava.net.URLClassLoader.findClass(Unknown Source)How to add a dependency into a jar file in java
What are the discoveries that have been possible with the rejection of positivism?
Trademark violation for app?
How could we fake a moon landing now?
The Nth Gryphon Number
Deconstruction is ambiguous
Drawing spherical mirrors
Are sorcerers unable to use the Careful Spell metamagic option on themselves?
How to save space when writing equations with cases?
Co-worker has annoying ringtone
A term for a woman complaining about things/begging in a cute/childish way
Do wooden building fires get hotter than 600°C?
If the probability of a dog barking one or more times in a given hour is 84%, then what is the probability of a dog barking in 30 minutes?
Significance of Cersei's obsession with elephants?
Why are vacuum tubes still used in amateur radios?
How can I prevent/balance waiting and turtling as a response to cooldown mechanics
Project Euler #1 in C++
How many time has Arya actually used Needle?
Time evolution of a Gaussian wave packet, why convert to k-space?
Can the Flaming Sphere spell be rammed into multiple Tiny creatures that are in the same 5-foot square?
Misunderstanding of Sylow theory
Strange behavior of Object.defineProperty() in JavaScript
How long can equipment go unused before powering up runs the risk of damage?
How much damage would a cupful of neutron star matter do to the Earth?
AppleTVs create a chatty alternate WiFi network
Cannot start kafka connect distributed using windows bat file
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
Data science time! April 2019 and salary with experience
The Ask Question Wizard is Live!Java Picasa API Directory listing not workingException in thread “main” java.lang.NoClassDefFoundError: org/apache/logging/log4j/message/MessageJDBC MySQL ConnectorGetting error in executing OWL APIHow to connect to mysql database using java?Unable to Setup CoAP Server using Californium Java LibraryHow to connect my Java applicationan to an Oracle databasespark kafka streaming Error - " java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtilsjava.net.URLClassLoader.findClass(Unknown Source)How to add a dependency into a jar file in java
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
0
UPDATE:
I have taken a look at the kafka.jar mentioned - which seems to be added to the classpath by the CMD files that run the distributed worker - kafka-run-class.bat in particular and it is less than 1kb so doesn't seem to be a valid JAR - so maybe that first error is a red herring. But that aside if it runs on, the distributed worker just spits out a load of errors about ClassNotFoundException, then goes on to say every config value provided in my distributed properties config file is not recognised, and doesn't try and load any plugins
ORIGINAL question
I am simply trying a very simple test of starting a Kafka Connect distributed worker, no plugins yet - just trying to get the basic worker going.
My distributed config is pretty stock
bootstrap.servers=GBV04303950:9092,GBV04303950:9082
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=2
config.storage.topic=connect-configs
config.storage.replication.factor=2
status.storage.topic=connect-status
status.storage.replication.factor=2
However I get the following error
C:kafkaClusterOneconnect-worker_1>.binwindowsconnect-distributed.bat .etckafka-connect-replicatorreplicator-connect-distributed.properties
[2019-03-22 10:53:51,832] WARN could not create Dir using jarFile from url file:/C:/kafka/ClusterOne/connect-worker_1/share/java/kafka/kafka.jar. skip
ping. (org.reflections.Reflections)
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at org.reflections.vfs.Vfs$DefaultUrlTypes$1.createDir(Vfs.java:216)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:99)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
at org.reflections.Reflections.scan(Reflections.java:240)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:412)
at org.reflections.Reflections$1.run(Reflections.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The only reason I can think of for this is my local drive is locked down and doesn't let you create folders unless you have Elevated rights. I am running this from a CMD prompt that was started with elevated rights but as I've noticed, this doesn't always translate down to passing those rights on to processes spawned from within the CMD prompt. But I'm not sure how I can provide elevated rights to the java process that's trying to unpack the JAR.
Also, that error is followed by a long stream of ClassNotFoundException, I assume because the above error means JARs cannot be unpacked to load all the required classes
eg
[2019-03-22 10:56:04,567] WARN could not get type for name org.apache.kafka.common.utils.MockTime from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.common.utils.MockTime
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.MockTime
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,706] WARN could not get type for name com.google.gson.JsonDeserializer from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.google.gson.JsonDeserializer
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonDeserializer
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,857] WARN could not get type for name org.scalatest.junit.JUnitSuite from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.junit.JUnitSuite
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.scalatest.junit.JUnitSuite
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
Can anyone suggest if they think my theory about elevated rights is why this isn't working or how I could get around this?
java apache-kafka apache-kafka-connect confluent
edited Mar 22 at 19:17
cricket_007
84.6k1147120
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
add a comment |
0
UPDATE:
I have taken a look at the kafka.jar mentioned - which seems to be added to the classpath by the CMD files that run the distributed worker - kafka-run-class.bat in particular and it is less than 1kb so doesn't seem to be a valid JAR - so maybe that first error is a red herring. But that aside if it runs on, the distributed worker just spits out a load of errors about ClassNotFoundException, then goes on to say every config value provided in my distributed properties config file is not recognised, and doesn't try and load any plugins
ORIGINAL question
I am simply trying a very simple test of starting a Kafka Connect distributed worker, no plugins yet - just trying to get the basic worker going.
My distributed config is pretty stock
bootstrap.servers=GBV04303950:9092,GBV04303950:9082
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=2
config.storage.topic=connect-configs
config.storage.replication.factor=2
status.storage.topic=connect-status
status.storage.replication.factor=2
However I get the following error
C:kafkaClusterOneconnect-worker_1>.binwindowsconnect-distributed.bat .etckafka-connect-replicatorreplicator-connect-distributed.properties
[2019-03-22 10:53:51,832] WARN could not create Dir using jarFile from url file:/C:/kafka/ClusterOne/connect-worker_1/share/java/kafka/kafka.jar. skip
ping. (org.reflections.Reflections)
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at org.reflections.vfs.Vfs$DefaultUrlTypes$1.createDir(Vfs.java:216)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:99)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
at org.reflections.Reflections.scan(Reflections.java:240)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:412)
at org.reflections.Reflections$1.run(Reflections.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The only reason I can think of for this is my local drive is locked down and doesn't let you create folders unless you have Elevated rights. I am running this from a CMD prompt that was started with elevated rights but as I've noticed, this doesn't always translate down to passing those rights on to processes spawned from within the CMD prompt. But I'm not sure how I can provide elevated rights to the java process that's trying to unpack the JAR.
Also, that error is followed by a long stream of ClassNotFoundException, I assume because the above error means JARs cannot be unpacked to load all the required classes
eg
[2019-03-22 10:56:04,567] WARN could not get type for name org.apache.kafka.common.utils.MockTime from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.common.utils.MockTime
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.MockTime
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,706] WARN could not get type for name com.google.gson.JsonDeserializer from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.google.gson.JsonDeserializer
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonDeserializer
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,857] WARN could not get type for name org.scalatest.junit.JUnitSuite from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.junit.JUnitSuite
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.scalatest.junit.JUnitSuite
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
Can anyone suggest if they think my theory about elevated rights is why this isn't working or how I could get around this?
java apache-kafka apache-kafka-connect confluent
edited Mar 22 at 19:17
cricket_007
84.6k1147120
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
Note: Windows is not supported by Confluent Platform, so running Replicator there probably isn't a good idea? Also, you would need to set theplugin.pathproperty to be the full path to theshare/javafolder, and there arent any folders within there that Connect would try to create, only read.
– cricket_007
Mar 22 at 19:18
add a comment |
0
0
0
UPDATE:
I have taken a look at the kafka.jar mentioned - which seems to be added to the classpath by the CMD files that run the distributed worker - kafka-run-class.bat in particular and it is less than 1kb so doesn't seem to be a valid JAR - so maybe that first error is a red herring. But that aside if it runs on, the distributed worker just spits out a load of errors about ClassNotFoundException, then goes on to say every config value provided in my distributed properties config file is not recognised, and doesn't try and load any plugins
ORIGINAL question
I am simply trying a very simple test of starting a Kafka Connect distributed worker, no plugins yet - just trying to get the basic worker going.
My distributed config is pretty stock
bootstrap.servers=GBV04303950:9092,GBV04303950:9082
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=2
config.storage.topic=connect-configs
config.storage.replication.factor=2
status.storage.topic=connect-status
status.storage.replication.factor=2
However I get the following error
C:kafkaClusterOneconnect-worker_1>.binwindowsconnect-distributed.bat .etckafka-connect-replicatorreplicator-connect-distributed.properties
[2019-03-22 10:53:51,832] WARN could not create Dir using jarFile from url file:/C:/kafka/ClusterOne/connect-worker_1/share/java/kafka/kafka.jar. skip
ping. (org.reflections.Reflections)
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at org.reflections.vfs.Vfs$DefaultUrlTypes$1.createDir(Vfs.java:216)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:99)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
at org.reflections.Reflections.scan(Reflections.java:240)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:412)
at org.reflections.Reflections$1.run(Reflections.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The only reason I can think of for this is my local drive is locked down and doesn't let you create folders unless you have Elevated rights. I am running this from a CMD prompt that was started with elevated rights but as I've noticed, this doesn't always translate down to passing those rights on to processes spawned from within the CMD prompt. But I'm not sure how I can provide elevated rights to the java process that's trying to unpack the JAR.
Also, that error is followed by a long stream of ClassNotFoundException, I assume because the above error means JARs cannot be unpacked to load all the required classes
eg
[2019-03-22 10:56:04,567] WARN could not get type for name org.apache.kafka.common.utils.MockTime from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.common.utils.MockTime
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.MockTime
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,706] WARN could not get type for name com.google.gson.JsonDeserializer from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.google.gson.JsonDeserializer
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonDeserializer
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,857] WARN could not get type for name org.scalatest.junit.JUnitSuite from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.junit.JUnitSuite
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.scalatest.junit.JUnitSuite
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
Can anyone suggest if they think my theory about elevated rights is why this isn't working or how I could get around this?
java apache-kafka apache-kafka-connect confluent
edited Mar 22 at 19:17
cricket_007
84.6k1147120
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
UPDATE:
I have taken a look at the kafka.jar mentioned - which seems to be added to the classpath by the CMD files that run the distributed worker - kafka-run-class.bat in particular and it is less than 1kb so doesn't seem to be a valid JAR - so maybe that first error is a red herring. But that aside if it runs on, the distributed worker just spits out a load of errors about ClassNotFoundException, then goes on to say every config value provided in my distributed properties config file is not recognised, and doesn't try and load any plugins
ORIGINAL question
I am simply trying a very simple test of starting a Kafka Connect distributed worker, no plugins yet - just trying to get the basic worker going.
My distributed config is pretty stock
bootstrap.servers=GBV04303950:9092,GBV04303950:9082
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=2
config.storage.topic=connect-configs
config.storage.replication.factor=2
status.storage.topic=connect-status
status.storage.replication.factor=2
However I get the following error
C:kafkaClusterOneconnect-worker_1>.binwindowsconnect-distributed.bat .etckafka-connect-replicatorreplicator-connect-distributed.properties
[2019-03-22 10:53:51,832] WARN could not create Dir using jarFile from url file:/C:/kafka/ClusterOne/connect-worker_1/share/java/kafka/kafka.jar. skip
ping. (org.reflections.Reflections)
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.zip.ZipFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at java.util.jar.JarFile.<init>(Unknown Source)
at org.reflections.vfs.Vfs$DefaultUrlTypes$1.createDir(Vfs.java:216)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:99)
at org.reflections.vfs.Vfs.fromURL(Vfs.java:91)
at org.reflections.Reflections.scan(Reflections.java:240)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.scan(DelegatingClassLoader.java:412)
at org.reflections.Reflections$1.run(Reflections.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
The only reason I can think of for this is my local drive is locked down and doesn't let you create folders unless you have Elevated rights. I am running this from a CMD prompt that was started with elevated rights but as I've noticed, this doesn't always translate down to passing those rights on to processes spawned from within the CMD prompt. But I'm not sure how I can provide elevated rights to the java process that's trying to unpack the JAR.
Also, that error is followed by a long stream of ClassNotFoundException, I assume because the above error means JARs cannot be unpacked to load all the required classes
eg
[2019-03-22 10:56:04,567] WARN could not get type for name org.apache.kafka.common.utils.MockTime from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.common.utils.MockTime
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.MockTime
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,706] WARN could not get type for name com.google.gson.JsonDeserializer from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name com.google.gson.JsonDeserializer
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonDeserializer
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
[2019-03-22 10:56:04,857] WARN could not get type for name org.scalatest.junit.JUnitSuite from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.junit.JUnitSuite
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader$InternalReflections.<init>(DelegatingClassLoader.java:404)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:304)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:242)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:183)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:61)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:90)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
Caused by: java.lang.ClassNotFoundException: org.scalatest.junit.JUnitSuite
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 10 more
Can anyone suggest if they think my theory about elevated rights is why this isn't working or how I could get around this?
java apache-kafka apache-kafka-connect confluent
java apache-kafka apache-kafka-connect confluent
edited Mar 22 at 19:17
cricket_007
84.6k1147120
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
edited Mar 22 at 19:17
cricket_007
84.6k1147120
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
edited Mar 22 at 19:17
cricket_007
84.6k1147120
edited Mar 22 at 19:17
cricket_007
84.6k1147120
edited Mar 22 at 19:17
cricket_007
84.6k1147120
84.6k1147120
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
asked Mar 22 at 11:00
NZJamesNZJames
2,04072958
2,04072958
Note: Windows is not supported by Confluent Platform, so running Replicator there probably isn't a good idea? Also, you would need to set theplugin.pathproperty to be the full path to theshare/javafolder, and there arent any folders within there that Connect would try to create, only read.
– cricket_007
Mar 22 at 19:18
add a comment |
Note: Windows is not supported by Confluent Platform, so running Replicator there probably isn't a good idea? Also, you would need to set theplugin.pathproperty to be the full path to theshare/javafolder, and there arent any folders within there that Connect would try to create, only read.
– cricket_007
Mar 22 at 19:18
Note: Windows is not supported by Confluent Platform, so running Replicator there probably isn't a good idea? Also, you would need to set the
plugin.path property to be the full path to the share/java folder, and there arent any folders within there that Connect would try to create, only read.– cricket_007
Mar 22 at 19:18
Note: Windows is not supported by Confluent Platform, so running Replicator there probably isn't a good idea? Also, you would need to set the
plugin.path property to be the full path to the share/java folder, and there arent any folders within there that Connect would try to create, only read.– cricket_007
Mar 22 at 19:18
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55298189%2fcannot-start-kafka-connect-distributed-using-windows-bat-file%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55298189%2fcannot-start-kafka-connect-distributed-using-windows-bat-file%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Note: Windows is not supported by Confluent Platform, so running Replicator there probably isn't a good idea? Also, you would need to set the
plugin.pathproperty to be the full path to theshare/javafolder, and there arent any folders within there that Connect would try to create, only read.– cricket_007
Mar 22 at 19:18