Styjun

Why most published works in medical imaging try reducing false positives?

Why did the person in charge of a principality not just declare themself king?

Why isn't 'chemically-strengthened glass' made with potassium carbonate to begin with?

How to patch glass cuts in a bicycle tire?

Construct a word ladder

Need to read my home electrical meter

Why does this if-statement combining assignment and an equality check return true?

Find the three digit Prime number P from the given unusual relationships

Apt - strange requests to d16r8ew072anqo.cloudfront.net:80

Can I connect my older mathematica front-end to the free wolfram engine?

My employer faked my resume to acquire projects

Where have Brexit voters gone?

How to cut a climbing rope?

Why aren't space telescopes put in GEO?

Is "cool" appropriate or offensive to use in IMs?

Can a person survive on blood in place of water?

NIntegrate doesn't evaluate

My players want to grind XP but we're using milestone advancement

Is the field of q-series 'dead'?

Using credit/debit card details vs swiping a card in a payment (credit card) terminal

When the Torah was almost lost and one (or several) Rabbis saved it?

Could a 19.25mm revolver actually exist?

How can I tell if I'm being too picky as a referee?

Defining the standard model of PA so that a space alien could understand

Write List of Map data into csv

Get item in the list in Scala?Scala list concatenation, ::: vs ++Apache Spark: map vs mapPartitions?What is the difference between map and flatMap and a good use case for each?Writing a RDD to a csvWrite single CSV file using spark-csvWriting an RDD to a CSV FileHow to create xml attributes dynamic via Map in scalaCase class mapping to csvHow to flatten RDD which contains sub list into main list

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

-1

1

val rdd = df.rdd.map(line => Row.fromSeq((
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem =>
 elem.label == "name1" 
 || elem.label == "name2" 
 || elem.label == "name3" 
 || elem.label == "name4" 

 ).map(elem => (elem.label -> elem.text)).toList)
 )

I do rdd.take(10).foreach(println), My is RDD[Row] then produced the output something like this:

[(name1, value1), (name2, value2),(name3, value3)]
[(name1, value11), (name2, value22),(name3, value33)]
[(name1, value111), (name2, value222),(name4, value44)]

I want save this into csv with (name1..name4 are header of csv), anyone please help how can I implement this with apache spark 2.4.0

name1 | name2 | name3 | name4
value1 | value2 |value3 | null
value11 | value22 |value33 | null
value111 | value222 |null | value444

scala apache-spark scala-xml

share|improve this question

edited Mar 24 at 5:25

kn3l

asked Mar 24 at 2:05

kn3lkn3l

7,998216599

add a comment | 

-1

1

val rdd = df.rdd.map(line => Row.fromSeq((
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem =>
 elem.label == "name1" 
 || elem.label == "name2" 
 || elem.label == "name3" 
 || elem.label == "name4" 

 ).map(elem => (elem.label -> elem.text)).toList)
 )

I do rdd.take(10).foreach(println), My is RDD[Row] then produced the output something like this:

[(name1, value1), (name2, value2),(name3, value3)]
[(name1, value11), (name2, value22),(name3, value33)]
[(name1, value111), (name2, value222),(name4, value44)]

I want save this into csv with (name1..name4 are header of csv), anyone please help how can I implement this with apache spark 2.4.0

name1 | name2 | name3 | name4
value1 | value2 |value3 | null
value11 | value22 |value33 | null
value111 | value222 |null | value444

scala apache-spark scala-xml

share|improve this question

edited Mar 24 at 5:25

kn3l

asked Mar 24 at 2:05

kn3lkn3l

7,998216599

add a comment | 

val rdd = df.rdd.map(line => Row.fromSeq((
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem =>
 elem.label == "name1" 
 || elem.label == "name2" 
 || elem.label == "name3" 
 || elem.label == "name4" 

 ).map(elem => (elem.label -> elem.text)).toList)
 )

I do rdd.take(10).foreach(println), My is RDD[Row] then produced the output something like this:

[(name1, value1), (name2, value2),(name3, value3)]
[(name1, value11), (name2, value22),(name3, value33)]
[(name1, value111), (name2, value222),(name4, value44)]

I want save this into csv with (name1..name4 are header of csv), anyone please help how can I implement this with apache spark 2.4.0

name1 | name2 | name3 | name4
value1 | value2 |value3 | null
value11 | value22 |value33 | null
value111 | value222 |null | value444

scala apache-spark scala-xml

share|improve this question

edited Mar 24 at 5:25

kn3l

asked Mar 24 at 2:05

kn3lkn3l

7,998216599

val rdd = df.rdd.map(line => Row.fromSeq((
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem =>
 elem.label == "name1" 
 || elem.label == "name2" 
 || elem.label == "name3" 
 || elem.label == "name4" 

 ).map(elem => (elem.label -> elem.text)).toList)
 )

[(name1, value1), (name2, value2),(name3, value3)]
[(name1, value11), (name2, value22),(name3, value33)]
[(name1, value111), (name2, value222),(name4, value44)]

name1 | name2 | name3 | name4
value1 | value2 |value3 | null
value11 | value22 |value33 | null
value111 | value222 |null | value444

share|improve this question

edited Mar 24 at 5:25

kn3l

asked Mar 24 at 2:05

kn3lkn3l

7,998216599

share|improve this question

edited Mar 24 at 5:25

kn3l

asked Mar 24 at 2:05

kn3lkn3l

7,998216599

asked Mar 24 at 2:05

kn3lkn3l

7,998216599

asked Mar 24 at 2:05

kn3lkn3l

7,998216599

1 Answer
1

active

oldest

votes

2

I adjusted your example and added some intermediate Values to help get each step:

 // define the labels you want:
 val labels = Seq("name1", "name2", "name3", "name4")
 val result: RDD[Row] = rdd.map line =>
 // your raw data
 val tuples: immutable.Seq[(String, String)] = 
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem => labels.contains(elem.label)) // you can use the label list to filter
 .map(elem => (elem.label -> elem.text)).toList // no change here
 val values: Seq[String] = 
 labels.map(l =>
 // take the values you have a label 
 tuples.findcase (k, v) => k == l.map(_._2)
 // or just add an empty String
 .getOrElse(""))
 // create a Row
 Row.fromSeq(values)
 

Now I am not sure - but in essence you have to insert the title Row as the first row:

[name1, name2, name3]

share|improve this answer

edited Mar 25 at 11:04

answered Mar 24 at 9:16

pmepme

4,08011935

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  that is not right, because when the name1 for example is missing from xml , how are we going to handle , how the header and data row going to consistent ? can you please help to provide the full of your solutions?
  
  – kn3l
  Mar 25 at 0:13
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  see my new answer - hope that gets you closer;)
  
  – pme
  Mar 25 at 11:04
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  thank for your response. I am really appreciated . however I found a solution on transform this to json dataset then I write to csv, so the table header will work accordingly .
  
  – kn3l
  Mar 26 at 4:07
  
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55320113%2fwrite-list-of-map-data-into-csv%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

I adjusted your example and added some intermediate Values to help get each step:

 // define the labels you want:
 val labels = Seq("name1", "name2", "name3", "name4")
 val result: RDD[Row] = rdd.map line =>
 // your raw data
 val tuples: immutable.Seq[(String, String)] = 
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem => labels.contains(elem.label)) // you can use the label list to filter
 .map(elem => (elem.label -> elem.text)).toList // no change here
 val values: Seq[String] = 
 labels.map(l =>
 // take the values you have a label 
 tuples.findcase (k, v) => k == l.map(_._2)
 // or just add an empty String
 .getOrElse(""))
 // create a Row
 Row.fromSeq(values)
 

Now I am not sure - but in essence you have to insert the title Row as the first row:

[name1, name2, name3]

share|improve this answer

edited Mar 25 at 11:04

answered Mar 24 at 9:16

pmepme

4,08011935

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  that is not right, because when the name1 for example is missing from xml , how are we going to handle , how the header and data row going to consistent ? can you please help to provide the full of your solutions?
  
  – kn3l
  Mar 25 at 0:13
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  see my new answer - hope that gets you closer;)
  
  – pme
  Mar 25 at 11:04
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  thank for your response. I am really appreciated . however I found a solution on transform this to json dataset then I write to csv, so the table header will work accordingly .
  
  – kn3l
  Mar 26 at 4:07
  
  

  
  
  
  

add a comment | 

2

I adjusted your example and added some intermediate Values to help get each step:

 // define the labels you want:
 val labels = Seq("name1", "name2", "name3", "name4")
 val result: RDD[Row] = rdd.map line =>
 // your raw data
 val tuples: immutable.Seq[(String, String)] = 
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem => labels.contains(elem.label)) // you can use the label list to filter
 .map(elem => (elem.label -> elem.text)).toList // no change here
 val values: Seq[String] = 
 labels.map(l =>
 // take the values you have a label 
 tuples.findcase (k, v) => k == l.map(_._2)
 // or just add an empty String
 .getOrElse(""))
 // create a Row
 Row.fromSeq(values)
 

Now I am not sure - but in essence you have to insert the title Row as the first row:

[name1, name2, name3]

share|improve this answer

edited Mar 25 at 11:04

answered Mar 24 at 9:16

pmepme

4,08011935

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  that is not right, because when the name1 for example is missing from xml , how are we going to handle , how the header and data row going to consistent ? can you please help to provide the full of your solutions?
  
  – kn3l
  Mar 25 at 0:13
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  see my new answer - hope that gets you closer;)
  
  – pme
  Mar 25 at 11:04
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  thank for your response. I am really appreciated . however I found a solution on transform this to json dataset then I write to csv, so the table header will work accordingly .
  
  – kn3l
  Mar 26 at 4:07
  
  

  
  
  
  

add a comment | 

I adjusted your example and added some intermediate Values to help get each step:

 // define the labels you want:
 val labels = Seq("name1", "name2", "name3", "name4")
 val result: RDD[Row] = rdd.map line =>
 // your raw data
 val tuples: immutable.Seq[(String, String)] = 
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem => labels.contains(elem.label)) // you can use the label list to filter
 .map(elem => (elem.label -> elem.text)).toList // no change here
 val values: Seq[String] = 
 labels.map(l =>
 // take the values you have a label 
 tuples.findcase (k, v) => k == l.map(_._2)
 // or just add an empty String
 .getOrElse(""))
 // create a Row
 Row.fromSeq(values)
 

Now I am not sure - but in essence you have to insert the title Row as the first row:

[name1, name2, name3]

share|improve this answer

edited Mar 25 at 11:04

answered Mar 24 at 9:16

pmepme

4,08011935

 // define the labels you want:
 val labels = Seq("name1", "name2", "name3", "name4")
 val result: RDD[Row] = rdd.map line =>
 // your raw data
 val tuples: immutable.Seq[(String, String)] = 
 scala.xml.XML.loadString("<?xml version='1.0' encoding='utf-8'?>" + line(1)).child
 .filter(elem => labels.contains(elem.label)) // you can use the label list to filter
 .map(elem => (elem.label -> elem.text)).toList // no change here
 val values: Seq[String] = 
 labels.map(l =>
 // take the values you have a label 
 tuples.findcase (k, v) => k == l.map(_._2)
 // or just add an empty String
 .getOrElse(""))
 // create a Row
 Row.fromSeq(values)
 

[name1, name2, name3]

share|improve this answer

edited Mar 25 at 11:04

answered Mar 24 at 9:16

pmepme

4,08011935

answered Mar 24 at 9:16

pmepme

4,08011935

answered Mar 24 at 9:16

pmepme

4,08011935