PySpark: Convert RDD to List The Next CEO of Stack OverflowHow do I check if a list is empty?Finding the index of an item given a list containing it in PythonConvert two lists into a dictionary in PythonDifference between append vs. extend list methods in PythonHow do you split a list into evenly sized chunks?Getting the last element of a list in PythonHow to make a flat list out of list of lists?How do I get the number of elements in a list in Python?How do I concatenate two lists in Python?How to clone or copy a list?

Is it possible to replace duplicates of a character with one character using tr

Does soap repel water?

Some questions about different axiomatic systems for neighbourhoods

Why don't programming languages automatically manage the synchronous/asynchronous problem?

Is there always a complete, orthogonal set of unitary matrices?

Is it okay to majorly distort historical facts while writing a fiction story?

How to count occurrences of text in a file?

Math-accent symbol over parentheses enclosing accented symbol (amsmath)

If Nick Fury and Coulson already knew about aliens (Kree and Skrull) why did they wait until Thor's appearance to start making weapons?

Is micro rebar a better way to reinforce concrete than rebar?

Bartok - Syncopation (1): Meaning of notes in between Grand Staff

What happened in Rome, when the western empire "fell"?

Is it my responsibility to learn a new technology in my own time my employer wants to implement?

Make solar eclipses exceedingly rare, but still have new moons

Is a distribution that is normal, but highly skewed considered Gaussian?

Won the lottery - how do I keep the money?

Is it convenient to ask the journal's editor for two additional days to complete a review?

Method for adding error messages to a dictionary given a key

Easy to read palindrome checker

Why doesn't UK go for the same deal Japan has with EU to resolve Brexit?

Why isn't acceleration always zero whenever velocity is zero, such as the moment a ball bounces off a wall?

Does increasing your ability score affect your main stat?

Why does standard notation not preserve intervals (visually)

Help understanding this unsettling image of Titan, Epimetheus, and Saturn's rings?



PySpark: Convert RDD to List



The Next CEO of Stack OverflowHow do I check if a list is empty?Finding the index of an item given a list containing it in PythonConvert two lists into a dictionary in PythonDifference between append vs. extend list methods in PythonHow do you split a list into evenly sized chunks?Getting the last element of a list in PythonHow to make a flat list out of list of lists?How do I get the number of elements in a list in Python?How do I concatenate two lists in Python?How to clone or copy a list?










0















I am having trouble converting an RDD to a list, and I could use some help seeing where I am going wrong.



Here is what I am working with:



This RDD has 49995 elements, and was created using this function:



testList = myData.map(extract_values)

testList.take(1)

[[SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0)]]


The extract_values function is:



def extract_values(friendRDD):
 list = []
 list.append(friendRDD[1])
 return list


At this point, I have tried:



myList = myData.map(extract_values).collect()


but it gives an error: 



ValueError: invalid literal for int() with base 10: ''


which I do not have any clue on why it is giving this error output.



How can I convert the testList RDD into a list at this point?



Here is the myData.take(1):



 [(0, SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0))],


Thank you for your help!






list pyspark rdd







share|improve this question
























  • It isn't necessarily a source of error since friendRDD could be anything, but just in case, I want to check to make sure you know Python list indices begin at zero.

    – Jesse Amano
    Mar 21 at 21:54











  • Since you're able to take an element from your RDD after mapping, you should be able to collect them all in the same way. The likely reason for this to be a problem would be if your source RDD contains some problematic rows and some non-problematic rows, and take(1) happened to choose a non-problematic row.

    – Jesse Amano
    Mar 21 at 22:03











  • To diagnose, check the source data (your myData RDD) for any "weird" records. Based on what the ValueError says, I'd especially keep an eye out for any vector fields that have an empty (or other non-integer) key.

    – Jesse Amano
    Mar 21 at 22:05











  • @Jesse Amano I understand that the python indices begin at 0. I'll post the myData take(1) above. I intended to take the [1] element of the vector.

    – MitterHai
    Mar 22 at 5:28












  • Ok, just checking ;) I still think you might have some rows in the RDD that don't have the exact same schema as the first row, since take(1) worked with that row but collect() didn't. There aren't any syntactical errors in what you've posted here, so it must be an issue with the data source.

    – Jesse Amano
    Mar 22 at 16:42















0















I am having trouble converting an RDD to a list, and I could use some help seeing where I am going wrong.



Here is what I am working with:



This RDD has 49995 elements, and was created using this function:



testList = myData.map(extract_values)

testList.take(1)

[[SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0)]]


The extract_values function is:



def extract_values(friendRDD):
 list = []
 list.append(friendRDD[1])
 return list


At this point, I have tried:



myList = myData.map(extract_values).collect()


but it gives an error: 



ValueError: invalid literal for int() with base 10: ''


which I do not have any clue on why it is giving this error output.



How can I convert the testList RDD into a list at this point?



Here is the myData.take(1):



 [(0, SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0))],


Thank you for your help!






list pyspark rdd







share|improve this question
























  • It isn't necessarily a source of error since friendRDD could be anything, but just in case, I want to check to make sure you know Python list indices begin at zero.

    – Jesse Amano
    Mar 21 at 21:54











  • Since you're able to take an element from your RDD after mapping, you should be able to collect them all in the same way. The likely reason for this to be a problem would be if your source RDD contains some problematic rows and some non-problematic rows, and take(1) happened to choose a non-problematic row.

    – Jesse Amano
    Mar 21 at 22:03











  • To diagnose, check the source data (your myData RDD) for any "weird" records. Based on what the ValueError says, I'd especially keep an eye out for any vector fields that have an empty (or other non-integer) key.

    – Jesse Amano
    Mar 21 at 22:05











  • @Jesse Amano I understand that the python indices begin at 0. I'll post the myData take(1) above. I intended to take the [1] element of the vector.

    – MitterHai
    Mar 22 at 5:28












  • Ok, just checking ;) I still think you might have some rows in the RDD that don't have the exact same schema as the first row, since take(1) worked with that row but collect() didn't. There aren't any syntactical errors in what you've posted here, so it must be an issue with the data source.

    – Jesse Amano
    Mar 22 at 16:42













0












0








0








I am having trouble converting an RDD to a list, and I could use some help seeing where I am going wrong.



Here is what I am working with:



This RDD has 49995 elements, and was created using this function:



testList = myData.map(extract_values)

testList.take(1)

[[SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0)]]


The extract_values function is:



def extract_values(friendRDD):
 list = []
 list.append(friendRDD[1])
 return list


At this point, I have tried:



myList = myData.map(extract_values).collect()


but it gives an error: 



ValueError: invalid literal for int() with base 10: ''


which I do not have any clue on why it is giving this error output.



How can I convert the testList RDD into a list at this point?



Here is the myData.take(1):



 [(0, SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0))],


Thank you for your help!






list pyspark rdd







share|improve this question
















I am having trouble converting an RDD to a list, and I could use some help seeing where I am going wrong.



Here is what I am working with:



This RDD has 49995 elements, and was created using this function:



testList = myData.map(extract_values)

testList.take(1)

[[SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0)]]


The extract_values function is:



def extract_values(friendRDD):
 list = []
 list.append(friendRDD[1])
 return list


At this point, I have tried:



myList = myData.map(extract_values).collect()


but it gives an error: 



ValueError: invalid literal for int() with base 10: ''


which I do not have any clue on why it is giving this error output.



How can I convert the testList RDD into a list at this point?



Here is the myData.take(1):



 [(0, SparseVector(49995, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 1.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0, 10: 1.0, 11: 1.0, 12: 1.0, 13: 1.0, 14: 1.0, 15: 1.0, 16: 1.0, 17: 1.0, 18: 1.0, 19: 1.0, 20: 1.0, 21: 1.0, 22: 1.0, 23: 1.0, 24: 1.0, 25: 1.0, 26: 1.0, 27: 1.0, 28: 1.0, 29: 1.0, 30: 1.0, 31: 1.0, 32: 1.0, 33: 1.0, 34: 1.0, 35: 1.0, 36: 1.0, 37: 1.0, 38: 1.0, 39: 1.0, 40: 1.0, 41: 1.0, 42: 1.0, 43: 1.0, 44: 1.0, 45: 1.0, 46: 1.0, 47: 1.0, 48: 1.0, 49: 1.0, 50: 1.0, 51: 1.0, 52: 1.0, 53: 1.0, 54: 1.0, 55: 1.0, 56: 1.0, 57: 1.0, 58: 1.0, 59: 1.0, 60: 1.0, 61: 1.0, 62: 1.0, 63: 1.0, 64: 1.0, 65: 1.0, 66: 1.0, 67: 1.0, 68: 1.0, 69: 1.0, 70: 1.0, 71: 1.0, 72: 1.0, 73: 1.0, 74: 1.0, 75: 1.0, 76: 1.0, 77: 1.0, 78: 1.0, 79: 1.0, 80: 1.0, 81: 1.0, 82: 1.0, 83: 1.0, 84: 1.0, 85: 1.0, 86: 1.0, 87: 1.0, 88: 1.0, 89: 1.0, 90: 1.0, 91: 1.0, 92: 1.0, 93: 1.0, 94: 1.0))],


Thank you for your help!






list pyspark rdd




list pyspark rdd






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 22 at 5:30







MitterHai

















asked Mar 21 at 18:06









MitterHaiMitterHai

145




145












  • It isn't necessarily a source of error since friendRDD could be anything, but just in case, I want to check to make sure you know Python list indices begin at zero.

    – Jesse Amano
    Mar 21 at 21:54











  • Since you're able to take an element from your RDD after mapping, you should be able to collect them all in the same way. The likely reason for this to be a problem would be if your source RDD contains some problematic rows and some non-problematic rows, and take(1) happened to choose a non-problematic row.

    – Jesse Amano
    Mar 21 at 22:03











  • To diagnose, check the source data (your myData RDD) for any "weird" records. Based on what the ValueError says, I'd especially keep an eye out for any vector fields that have an empty (or other non-integer) key.

    – Jesse Amano
    Mar 21 at 22:05











  • @Jesse Amano I understand that the python indices begin at 0. I'll post the myData take(1) above. I intended to take the [1] element of the vector.

    – MitterHai
    Mar 22 at 5:28












  • Ok, just checking ;) I still think you might have some rows in the RDD that don't have the exact same schema as the first row, since take(1) worked with that row but collect() didn't. There aren't any syntactical errors in what you've posted here, so it must be an issue with the data source.

    – Jesse Amano
    Mar 22 at 16:42

















  • It isn't necessarily a source of error since friendRDD could be anything, but just in case, I want to check to make sure you know Python list indices begin at zero.

    – Jesse Amano
    Mar 21 at 21:54











  • Since you're able to take an element from your RDD after mapping, you should be able to collect them all in the same way. The likely reason for this to be a problem would be if your source RDD contains some problematic rows and some non-problematic rows, and take(1) happened to choose a non-problematic row.

    – Jesse Amano
    Mar 21 at 22:03











  • To diagnose, check the source data (your myData RDD) for any "weird" records. Based on what the ValueError says, I'd especially keep an eye out for any vector fields that have an empty (or other non-integer) key.

    – Jesse Amano
    Mar 21 at 22:05











  • @Jesse Amano I understand that the python indices begin at 0. I'll post the myData take(1) above. I intended to take the [1] element of the vector.

    – MitterHai
    Mar 22 at 5:28












  • Ok, just checking ;) I still think you might have some rows in the RDD that don't have the exact same schema as the first row, since take(1) worked with that row but collect() didn't. There aren't any syntactical errors in what you've posted here, so it must be an issue with the data source.

    – Jesse Amano
    Mar 22 at 16:42
















It isn't necessarily a source of error since friendRDD could be anything, but just in case, I want to check to make sure you know Python list indices begin at zero.

– Jesse Amano
Mar 21 at 21:54





It isn't necessarily a source of error since friendRDD could be anything, but just in case, I want to check to make sure you know Python list indices begin at zero.

– Jesse Amano
Mar 21 at 21:54













Since you're able to take an element from your RDD after mapping, you should be able to collect them all in the same way. The likely reason for this to be a problem would be if your source RDD contains some problematic rows and some non-problematic rows, and take(1) happened to choose a non-problematic row.

– Jesse Amano
Mar 21 at 22:03





Since you're able to take an element from your RDD after mapping, you should be able to collect them all in the same way. The likely reason for this to be a problem would be if your source RDD contains some problematic rows and some non-problematic rows, and take(1) happened to choose a non-problematic row.

– Jesse Amano
Mar 21 at 22:03













To diagnose, check the source data (your myData RDD) for any "weird" records. Based on what the ValueError says, I'd especially keep an eye out for any vector fields that have an empty (or other non-integer) key.

– Jesse Amano
Mar 21 at 22:05





To diagnose, check the source data (your myData RDD) for any "weird" records. Based on what the ValueError says, I'd especially keep an eye out for any vector fields that have an empty (or other non-integer) key.

– Jesse Amano
Mar 21 at 22:05













@Jesse Amano I understand that the python indices begin at 0. I'll post the myData take(1) above. I intended to take the [1] element of the vector.

– MitterHai
Mar 22 at 5:28






@Jesse Amano I understand that the python indices begin at 0. I'll post the myData take(1) above. I intended to take the [1] element of the vector.

– MitterHai
Mar 22 at 5:28














Ok, just checking ;) I still think you might have some rows in the RDD that don't have the exact same schema as the first row, since take(1) worked with that row but collect() didn't. There aren't any syntactical errors in what you've posted here, so it must be an issue with the data source.

– Jesse Amano
Mar 22 at 16:42





Ok, just checking ;) I still think you might have some rows in the RDD that don't have the exact same schema as the first row, since take(1) worked with that row but collect() didn't. There aren't any syntactical errors in what you've posted here, so it must be an issue with the data source.

– Jesse Amano
Mar 22 at 16:42












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55286688%2fpyspark-convert-rdd-to-list%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55286688%2fpyspark-convert-rdd-to-list%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

How to preserve electronics (computers, tablets and phones) for hundreds of years Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) The network's official Twitter account is up and running again. What content…How might modern humans leave a message for 50,000 years?How would an aquatic race develop computers?How do we make computers care for human life?Keeping supplies for 1000 yearsWhat will cell phones become in 100 years?What body armor protects against a laser?Could a society attain 1800s technology if limited to using copper and bronze?Electricity power (magic) and electronicsScavenging metal resources in post-apocalypse after 100 yearsWhy would precursors create devices that can survive and still work after hundreds of years?

Image
Strange behavior of Object.defineProperty() in JavaScript How to save space when writing equations with cases? Which languages to learn for historical linguistics? Amount of permutations on an NxNxN Rubik's Cube The test team as an enemy of development? And how can this be avoided? One-one communication How does Belgium enforce obligatory attendance in elections? I can't produce songs What are the discoveries that have been possible with the rejection of positivism? An adverb for when you're not exaggerating Would it be easier to apply for a UK visa if there is a host family to sponsor for you in going there? What is the meaning of 'breadth' in breadth first search? Why we try to capture variability? What do you call the main part of a joke? Has negative voting ever been officially implemented in elections, or seriously proposed, or even studied? How can I prevent/balance waiting and turtling as a response to cooldown mechanics What...
Read more

155 수학 과학 기타 둘러보기 메뉴eh추가해eh문서를 완성해

Image
0123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369...
Read more

Convert JSON to MapSimplest way to convert a string representing JSON array into Java array List of key value pairsHow to take String JSON values into HashMapConvert a JSON String to a HashMapCreating BSON object from JSON stringJSONObject - How to get a value ?Better Map ConstructorReading JSON from Url javaUpdate columns if input values are not null otherwise ignore and keep the existing values of column in databaseHow to store object represented as Json string into Couchbase Lite?Go JSON into MapHow do I efficiently iterate over each entry in a Java Map?Sort a Map<Key, Value> by valuesHow to round a number to n decimal places in JavaCan comments be used in JSON?How do I read / convert an InputStream into a String in Java?How can I pretty-print JSON in a shell script?What is the correct JSON content type?How can I deserialize JSON to a simple Dictionary<string,string> in ASP.NET?Why does Google prepend while(1); to their JSON responses?Parse JSON in JavaScript?

Image
How to know if a folder is a symbolic link? Looking for a soft substance that doesn't dissolve underwater Is there a way to make it so the cursor is included when I prtscr key? If a person had control of every single cell of their body, would they be able to transform into another creature? Why do most published works in medical imaging try to reduce false positives? Why do Ryanair allow me to book connecting itineraries through a third party, but not through their own website? Does Nitrogen inside commercial airliner wheels prevent blowouts on touchdown? Are these reasonable traits for someone with autism? How to make a villain fall in love? Would jet fuel for an F-16 or F-35 be producible during WW2? Is there an efficient way to replace text matching the entire content of one file with the entire content of another file? Using credit/debit card details vs swiping a card in a payment (credit card) terminal Should I disclose a colleague's illness (that I...
Read more