Styjun

Simulate a 1D Game-of-Life-ish Model

Manager manipulates my leaves, what's in it for him?

Make Interviewee Comfortable in Potentially Intimate Environment

Do household ovens ventilate heat to the outdoors?

Norwegian refuses EU delay (4.7 hours) compensation because it turned out there was nothing wrong with the aircraft

Are lay articles good enough to be the main source of information for PhD research?

Which museums have artworks of all four ninja turtles' namesakes?

Is it true that, "just ten trading days represent 63 per cent of the returns of the past 50 years"?

How should I avoid someone patenting technology in my paper/poster?

How does one calculate the distribution of the Matt Colville way of rolling stats?

Minimize taxes now that I earn more

Debussy as term for bathroom?

Is it really necessary to have 4 hours meeting in Sprint planning?

Cheap antenna for new HF HAM

Automate tasks with Lambdas in java

What informations can we obtain with these voltage and current measurements of a little electronic device?

Temporarily moving a SQL Server 2016 database to SQL Server 2017 and then moving back. Is it possible?

Calibrated Esteps Causes Extruder Skipping

Was there a trial by combat between a man and a dog in medieval France?

Nanomachines exist that enable Axolotl-levels of regeneration - So how can crippling injuries exist as well?

Applications of mathematics in clinical setting

How to manage expenditure when billing cycles and paycheck cycles are not aligned?

Is there an in-universe reason Harry says this or is this simply a Rowling mistake?

How can I prevent soul energy from dissipating?

How to read large zip files in pyspark

How to open/stream .zip files through Spark?How to merge two dictionaries in a single expression?How do I check if a list is empty?How do I check whether a file exists without exceptions?How do I copy a file in Python?How can I safely create a nested directory?How do I sort a dictionary by value?How to make a chain of function decorators?How do you read from stdin?How do I list all files of a directory?How to read a file line-by-line into a list?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

I do have n number of .zip files on s3, which I want to process and extract some data out of them. zip files contains a single json file. In Spark we can read .gz files, but I didn't find any way to read data within .zip files. Can someone please help me out how can I process large zip files over spark using python. I came across some options like newAPIHadoopFile, but didn't get any luck with them, nor found way to implement them in pyspark. Please note the zip files are >1G, some are of 20G as well.

Below is code, which I used:

import zipfile
import io
file_name = "s3 file path for zip file"

def zip_extract(x):
 in_memory_data = io.BytesIO(x[1])
 file_obj = zipfile.ZipFile(in_memory_data, "r")
 files = [i for i in file_obj.namelist()]
 return dict(zip(files, [file_obj.open(file).read() for file in files]))


zips = sc.binaryFiles(file_name)
files_data = zips.map(zip_extract)

But it's failing because of below reason. The instance which I'm using is r42x.large.

Exit code: 52
Stack trace: ExitCodeException exitCode=52: 
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0

python pyspark amazon-emr

share|improve this question

edited Jul 8 at 14:02

thebluephantom

4,78255 gold badges1515 silver badges3737 bronze badges

asked Mar 28 at 14:29

SandieSandie

12111 gold badge22 silver badges1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Possible duplicate of How to open/stream .zip files through Spark?
  
  – Jim Todd
  Mar 28 at 17:06
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Had a look already, that is not working.
  
  – Sandie
  Mar 28 at 17:22
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Briefly add your code that you tried, and also the error you get. That would be great.
  
  – Jim Todd
  Mar 28 at 17:23
  

  
  
  
  

add a comment
 | 

1

I do have n number of .zip files on s3, which I want to process and extract some data out of them. zip files contains a single json file. In Spark we can read .gz files, but I didn't find any way to read data within .zip files. Can someone please help me out how can I process large zip files over spark using python. I came across some options like newAPIHadoopFile, but didn't get any luck with them, nor found way to implement them in pyspark. Please note the zip files are >1G, some are of 20G as well.

Below is code, which I used:

import zipfile
import io
file_name = "s3 file path for zip file"

def zip_extract(x):
 in_memory_data = io.BytesIO(x[1])
 file_obj = zipfile.ZipFile(in_memory_data, "r")
 files = [i for i in file_obj.namelist()]
 return dict(zip(files, [file_obj.open(file).read() for file in files]))


zips = sc.binaryFiles(file_name)
files_data = zips.map(zip_extract)

But it's failing because of below reason. The instance which I'm using is r42x.large.

Exit code: 52
Stack trace: ExitCodeException exitCode=52: 
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0

python pyspark amazon-emr

share|improve this question

edited Jul 8 at 14:02

thebluephantom

4,78255 gold badges1515 silver badges3737 bronze badges

asked Mar 28 at 14:29

SandieSandie

12111 gold badge22 silver badges1313 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Possible duplicate of How to open/stream .zip files through Spark?
  
  – Jim Todd
  Mar 28 at 17:06
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Had a look already, that is not working.
  
  – Sandie
  Mar 28 at 17:22
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Briefly add your code that you tried, and also the error you get. That would be great.
  
  – Jim Todd
  Mar 28 at 17:23
  

  
  
  
  

add a comment
 | 

I do have n number of .zip files on s3, which I want to process and extract some data out of them. zip files contains a single json file. In Spark we can read .gz files, but I didn't find any way to read data within .zip files. Can someone please help me out how can I process large zip files over spark using python. I came across some options like newAPIHadoopFile, but didn't get any luck with them, nor found way to implement them in pyspark. Please note the zip files are >1G, some are of 20G as well.

Below is code, which I used:

import zipfile
import io
file_name = "s3 file path for zip file"

def zip_extract(x):
 in_memory_data = io.BytesIO(x[1])
 file_obj = zipfile.ZipFile(in_memory_data, "r")
 files = [i for i in file_obj.namelist()]
 return dict(zip(files, [file_obj.open(file).read() for file in files]))


zips = sc.binaryFiles(file_name)
files_data = zips.map(zip_extract)

But it's failing because of below reason. The instance which I'm using is r42x.large.

Exit code: 52
Stack trace: ExitCodeException exitCode=52: 
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0

python pyspark amazon-emr

share|improve this question

edited Jul 8 at 14:02

thebluephantom

4,78255 gold badges1515 silver badges3737 bronze badges

asked Mar 28 at 14:29

SandieSandie

12111 gold badge22 silver badges1313 bronze badges

import zipfile
import io
file_name = "s3 file path for zip file"

def zip_extract(x):
 in_memory_data = io.BytesIO(x[1])
 file_obj = zipfile.ZipFile(in_memory_data, "r")
 files = [i for i in file_obj.namelist()]
 return dict(zip(files, [file_obj.open(file).read() for file in files]))


zips = sc.binaryFiles(file_name)
files_data = zips.map(zip_extract)

Exit code: 52
Stack trace: ExitCodeException exitCode=52: 
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0

share|improve this question

edited Jul 8 at 14:02

thebluephantom

4,78255 gold badges1515 silver badges3737 bronze badges

asked Mar 28 at 14:29

SandieSandie

12111 gold badge22 silver badges1313 bronze badges

share|improve this question

edited Jul 8 at 14:02

thebluephantom

4,78255 gold badges1515 silver badges3737 bronze badges

asked Mar 28 at 14:29

SandieSandie

12111 gold badge22 silver badges1313 bronze badges

edited Jul 8 at 14:02

thebluephantom

4,78255 gold badges1515 silver badges3737 bronze badges

edited Jul 8 at 14:02

thebluephantom

4,78255 gold badges1515 silver badges3737 bronze badges

asked Mar 28 at 14:29

SandieSandie

12111 gold badge22 silver badges1313 bronze badges

asked Mar 28 at 14:29

SandieSandie

12111 gold badge22 silver badges1313 bronze badges