Encountered an error executing step masterStep Execption Loading data to mysql Spring BatchSpring batch job throws error like 'Partition handler returned an unsuccessful step'spring boot :Injection of autowired dependencies failed;Adding maven dependency gives a specific error during my spring batchEncountering error during context initialization in springLazy load spring beanCannot load driver class: com.mysql.jdbc.Driver Error on Spring Boot using EclipseJob called twice in SpringBatch when configured with modular=trueUnable to start embedded Tomcat NoClassDefFoundError DispatcherServletPatherror:spring boot database error “org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration”spring boot database error datasource "org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration”Spring batch pass value of resource from step 1 to the next step
Importance of moon phases for Apollo missions
Can the caster of Time Stop still use their bonus action or reaction?
What should I watch before playing Alien: Isolation?
What does a Nintendo Game Boy do when turned on without a game cartridge inserted?
I have a domain, static IP and many devices I'd like to access outside my house. How to route them?
What do Unicorns want?
Is it better to have a 10 year gap or a bad reference?
Would using carbon dioxide as fuel work to reduce the greenhouse effect?
Did Don Young threaten John Boehner with a 10 inch blade to the throat?
what does the term highest qualification mean?
Why is the UH-60 tail rotor canted?
Why does airflow separate from the wing during stall?
Is there an English word to describe when a sound "protrudes"?
Cargo capacity of a kayak
What is the best word describing the nature of expiring in a short amount of time, connoting "losing public attention"?
Which dice game has a board with 9x9 squares that has different colors on the diagonals and midway on some edges?
Monday's Blocking Donimoes Problem
Why can't a country print its own money to spend it only abroad?
What is the metal bit in the front of this propeller spinner?
Reissue US, UK, Canada visas in stolen passports
What would be the effects of (relatively) widespread precognition on the stock market?
Oriented vector bundle with odd-dimensional fibers
"It is what it is"
Why was Quirrell said to be in the Black Forest if Voldemort was actually in Albania?
Encountered an error executing step masterStep Execption Loading data to mysql Spring Batch
Spring batch job throws error like 'Partition handler returned an unsuccessful step'spring boot :Injection of autowired dependencies failed;Adding maven dependency gives a specific error during my spring batchEncountering error during context initialization in springLazy load spring beanCannot load driver class: com.mysql.jdbc.Driver Error on Spring Boot using EclipseJob called twice in SpringBatch when configured with modular=trueUnable to start embedded Tomcat NoClassDefFoundError DispatcherServletPatherror:spring boot database error “org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration”spring boot database error datasource "org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration”Spring batch pass value of resource from step 1 to the next step
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I am trying to load set of csv file into mysql using spring batch.
My code is given below. But when I run with 2 or 3 files the data is loading fine. but when I try with 100 or more Its trowing an execption aftre loading 4 to 8 files data.
Exception:
ERROR 11633 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step masterStep in job importUserJob
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:66) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:136) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:797) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:781) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:137) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
I don't know whats happening behind, Any help appreciated.
Code:
@Configuration
@EnableBatchProcessing
public class ImportJobConfig
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private DataSource dataSource;
@Autowired
private JdbcBatchItemWriter<Person> writer;
@Autowired
private FlatFileItemReader<Person> personItemReader;
@Bean
public DataSource dataSource()
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("URL");
dataSource.setUsername("USERNAME");
dataSource.setPassword("PASSWORD");
return dataSource;
@Bean
public ThreadPoolTaskExecutor taskExecutor()
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(4);
taskExecutor.setCorePoolSize(4);
taskExecutor.setQueueCapacity(4);
taskExecutor.afterPropertiesSet();
taskExecutor.setAllowCoreThreadTimeOut(true);
return taskExecutor;
@Bean
@Qualifier("masterStep")
public Step masterStep()
return stepBuilderFactory.get("masterStep").partitioner("step1", partitioner()).step(step1())
.taskExecutor(taskExecutor()).build();
@Bean("partitioner")
@StepScope
public Partitioner partitioner()
System.out.println("In Partitioner");
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
String filePath = "//data//person//*.csv";
Resource[] resources = null;
try
resources = resolver.getResources("file:" + filePath);
catch (IOException e)
e.printStackTrace();
partitioner.setResources(resources);
partitioner.partition(4);
return partitioner;
@Bean
public APSUploadFileItemProcessor processor()
return new APSUploadFileItemProcessor();
@Bean
public JdbcBatchItemWriter<Person> writer()
JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<>();
try
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO person(name,business,phone) VALUES(name, :business, :phone)");
writer.setDataSource(dataSource);
catch (Exception e)
e.printStackTrace();
return writer;
@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1)
return jobBuilderFactory.get("importUserJob").incrementer(new RunIdIncrementer()).listener(listener)
.flow(masterStep()).end().build();
@Bean
public Step step1()
return stepBuilderFactory.get("step1").<Person, Person>chunk(10000).processor(processor())
.writer(writer).reader(personItemReader).build();
@Bean
@StepScope
@Qualifier("personItemReader")
@DependsOn("partitioner")
public FlatFileItemReader<Person> personItemReader(
@Value("#stepExecutionContext['fileName']") String filename) throws MalformedURLException ")
.names(new String[] "name","business","phone" )
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>()
setTargetType(Person.class);
).resource(new UrlResource(filename)).build();
java spring spring-boot spring-batch batch-processing
add a comment |
I am trying to load set of csv file into mysql using spring batch.
My code is given below. But when I run with 2 or 3 files the data is loading fine. but when I try with 100 or more Its trowing an execption aftre loading 4 to 8 files data.
Exception:
ERROR 11633 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step masterStep in job importUserJob
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:66) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:136) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:797) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:781) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:137) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
I don't know whats happening behind, Any help appreciated.
Code:
@Configuration
@EnableBatchProcessing
public class ImportJobConfig
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private DataSource dataSource;
@Autowired
private JdbcBatchItemWriter<Person> writer;
@Autowired
private FlatFileItemReader<Person> personItemReader;
@Bean
public DataSource dataSource()
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("URL");
dataSource.setUsername("USERNAME");
dataSource.setPassword("PASSWORD");
return dataSource;
@Bean
public ThreadPoolTaskExecutor taskExecutor()
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(4);
taskExecutor.setCorePoolSize(4);
taskExecutor.setQueueCapacity(4);
taskExecutor.afterPropertiesSet();
taskExecutor.setAllowCoreThreadTimeOut(true);
return taskExecutor;
@Bean
@Qualifier("masterStep")
public Step masterStep()
return stepBuilderFactory.get("masterStep").partitioner("step1", partitioner()).step(step1())
.taskExecutor(taskExecutor()).build();
@Bean("partitioner")
@StepScope
public Partitioner partitioner()
System.out.println("In Partitioner");
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
String filePath = "//data//person//*.csv";
Resource[] resources = null;
try
resources = resolver.getResources("file:" + filePath);
catch (IOException e)
e.printStackTrace();
partitioner.setResources(resources);
partitioner.partition(4);
return partitioner;
@Bean
public APSUploadFileItemProcessor processor()
return new APSUploadFileItemProcessor();
@Bean
public JdbcBatchItemWriter<Person> writer()
JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<>();
try
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO person(name,business,phone) VALUES(name, :business, :phone)");
writer.setDataSource(dataSource);
catch (Exception e)
e.printStackTrace();
return writer;
@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1)
return jobBuilderFactory.get("importUserJob").incrementer(new RunIdIncrementer()).listener(listener)
.flow(masterStep()).end().build();
@Bean
public Step step1()
return stepBuilderFactory.get("step1").<Person, Person>chunk(10000).processor(processor())
.writer(writer).reader(personItemReader).build();
@Bean
@StepScope
@Qualifier("personItemReader")
@DependsOn("partitioner")
public FlatFileItemReader<Person> personItemReader(
@Value("#stepExecutionContext['fileName']") String filename) throws MalformedURLException ")
.names(new String[] "name","business","phone" )
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>()
setTargetType(Person.class);
).resource(new UrlResource(filename)).build();
java spring spring-boot spring-batch batch-processing
1
Please post the full stack trace of the exception.
– Mahmoud Ben Hassine
Mar 26 at 14:21
this is the entire stack trace I'm getting
– Juhan
Mar 26 at 15:12
2
See > stackoverflow.com/questions/32629724/…
– Mountain-Lion
Mar 26 at 15:55
@Mountain-Lion I went through the question and made changes to the code, but still facing the same issue
– Juhan
Mar 27 at 2:23
add a comment |
I am trying to load set of csv file into mysql using spring batch.
My code is given below. But when I run with 2 or 3 files the data is loading fine. but when I try with 100 or more Its trowing an execption aftre loading 4 to 8 files data.
Exception:
ERROR 11633 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step masterStep in job importUserJob
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:66) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:136) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:797) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:781) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:137) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
I don't know whats happening behind, Any help appreciated.
Code:
@Configuration
@EnableBatchProcessing
public class ImportJobConfig
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private DataSource dataSource;
@Autowired
private JdbcBatchItemWriter<Person> writer;
@Autowired
private FlatFileItemReader<Person> personItemReader;
@Bean
public DataSource dataSource()
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("URL");
dataSource.setUsername("USERNAME");
dataSource.setPassword("PASSWORD");
return dataSource;
@Bean
public ThreadPoolTaskExecutor taskExecutor()
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(4);
taskExecutor.setCorePoolSize(4);
taskExecutor.setQueueCapacity(4);
taskExecutor.afterPropertiesSet();
taskExecutor.setAllowCoreThreadTimeOut(true);
return taskExecutor;
@Bean
@Qualifier("masterStep")
public Step masterStep()
return stepBuilderFactory.get("masterStep").partitioner("step1", partitioner()).step(step1())
.taskExecutor(taskExecutor()).build();
@Bean("partitioner")
@StepScope
public Partitioner partitioner()
System.out.println("In Partitioner");
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
String filePath = "//data//person//*.csv";
Resource[] resources = null;
try
resources = resolver.getResources("file:" + filePath);
catch (IOException e)
e.printStackTrace();
partitioner.setResources(resources);
partitioner.partition(4);
return partitioner;
@Bean
public APSUploadFileItemProcessor processor()
return new APSUploadFileItemProcessor();
@Bean
public JdbcBatchItemWriter<Person> writer()
JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<>();
try
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO person(name,business,phone) VALUES(name, :business, :phone)");
writer.setDataSource(dataSource);
catch (Exception e)
e.printStackTrace();
return writer;
@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1)
return jobBuilderFactory.get("importUserJob").incrementer(new RunIdIncrementer()).listener(listener)
.flow(masterStep()).end().build();
@Bean
public Step step1()
return stepBuilderFactory.get("step1").<Person, Person>chunk(10000).processor(processor())
.writer(writer).reader(personItemReader).build();
@Bean
@StepScope
@Qualifier("personItemReader")
@DependsOn("partitioner")
public FlatFileItemReader<Person> personItemReader(
@Value("#stepExecutionContext['fileName']") String filename) throws MalformedURLException ")
.names(new String[] "name","business","phone" )
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>()
setTargetType(Person.class);
).resource(new UrlResource(filename)).build();
java spring spring-boot spring-batch batch-processing
I am trying to load set of csv file into mysql using spring batch.
My code is given below. But when I run with 2 or 3 files the data is loading fine. but when I try with 100 or more Its trowing an execption aftre loading 4 to 8 files data.
Exception:
ERROR 11633 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step masterStep in job importUserJob
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:66) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:136) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:797) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:781) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:137) [spring-boot-2.0.1.RELEASE.jar!/:2.0.1.RELEASE]
I don't know whats happening behind, Any help appreciated.
Code:
@Configuration
@EnableBatchProcessing
public class ImportJobConfig
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private DataSource dataSource;
@Autowired
private JdbcBatchItemWriter<Person> writer;
@Autowired
private FlatFileItemReader<Person> personItemReader;
@Bean
public DataSource dataSource()
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("URL");
dataSource.setUsername("USERNAME");
dataSource.setPassword("PASSWORD");
return dataSource;
@Bean
public ThreadPoolTaskExecutor taskExecutor()
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(4);
taskExecutor.setCorePoolSize(4);
taskExecutor.setQueueCapacity(4);
taskExecutor.afterPropertiesSet();
taskExecutor.setAllowCoreThreadTimeOut(true);
return taskExecutor;
@Bean
@Qualifier("masterStep")
public Step masterStep()
return stepBuilderFactory.get("masterStep").partitioner("step1", partitioner()).step(step1())
.taskExecutor(taskExecutor()).build();
@Bean("partitioner")
@StepScope
public Partitioner partitioner()
System.out.println("In Partitioner");
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
String filePath = "//data//person//*.csv";
Resource[] resources = null;
try
resources = resolver.getResources("file:" + filePath);
catch (IOException e)
e.printStackTrace();
partitioner.setResources(resources);
partitioner.partition(4);
return partitioner;
@Bean
public APSUploadFileItemProcessor processor()
return new APSUploadFileItemProcessor();
@Bean
public JdbcBatchItemWriter<Person> writer()
JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<>();
try
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO person(name,business,phone) VALUES(name, :business, :phone)");
writer.setDataSource(dataSource);
catch (Exception e)
e.printStackTrace();
return writer;
@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1)
return jobBuilderFactory.get("importUserJob").incrementer(new RunIdIncrementer()).listener(listener)
.flow(masterStep()).end().build();
@Bean
public Step step1()
return stepBuilderFactory.get("step1").<Person, Person>chunk(10000).processor(processor())
.writer(writer).reader(personItemReader).build();
@Bean
@StepScope
@Qualifier("personItemReader")
@DependsOn("partitioner")
public FlatFileItemReader<Person> personItemReader(
@Value("#stepExecutionContext['fileName']") String filename) throws MalformedURLException ")
.names(new String[] "name","business","phone" )
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>()
setTargetType(Person.class);
).resource(new UrlResource(filename)).build();
java spring spring-boot spring-batch batch-processing
java spring spring-boot spring-batch batch-processing
asked Mar 26 at 13:24
JuhanJuhan
7601 gold badge5 silver badges21 bronze badges
7601 gold badge5 silver badges21 bronze badges
1
Please post the full stack trace of the exception.
– Mahmoud Ben Hassine
Mar 26 at 14:21
this is the entire stack trace I'm getting
– Juhan
Mar 26 at 15:12
2
See > stackoverflow.com/questions/32629724/…
– Mountain-Lion
Mar 26 at 15:55
@Mountain-Lion I went through the question and made changes to the code, but still facing the same issue
– Juhan
Mar 27 at 2:23
add a comment |
1
Please post the full stack trace of the exception.
– Mahmoud Ben Hassine
Mar 26 at 14:21
this is the entire stack trace I'm getting
– Juhan
Mar 26 at 15:12
2
See > stackoverflow.com/questions/32629724/…
– Mountain-Lion
Mar 26 at 15:55
@Mountain-Lion I went through the question and made changes to the code, but still facing the same issue
– Juhan
Mar 27 at 2:23
1
1
Please post the full stack trace of the exception.
– Mahmoud Ben Hassine
Mar 26 at 14:21
Please post the full stack trace of the exception.
– Mahmoud Ben Hassine
Mar 26 at 14:21
this is the entire stack trace I'm getting
– Juhan
Mar 26 at 15:12
this is the entire stack trace I'm getting
– Juhan
Mar 26 at 15:12
2
2
See > stackoverflow.com/questions/32629724/…
– Mountain-Lion
Mar 26 at 15:55
See > stackoverflow.com/questions/32629724/…
– Mountain-Lion
Mar 26 at 15:55
@Mountain-Lion I went through the question and made changes to the code, but still facing the same issue
– Juhan
Mar 27 at 2:23
@Mountain-Lion I went through the question and made changes to the code, but still facing the same issue
– Juhan
Mar 27 at 2:23
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55358288%2fencountered-an-error-executing-step-masterstep-execption-loading-data-to-mysql-s%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55358288%2fencountered-an-error-executing-step-masterstep-execption-loading-data-to-mysql-s%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Please post the full stack trace of the exception.
– Mahmoud Ben Hassine
Mar 26 at 14:21
this is the entire stack trace I'm getting
– Juhan
Mar 26 at 15:12
2
See > stackoverflow.com/questions/32629724/…
– Mountain-Lion
Mar 26 at 15:55
@Mountain-Lion I went through the question and made changes to the code, but still facing the same issue
– Juhan
Mar 27 at 2:23