How to save a json from dataframe with specific format?How do I format a Microsoft JSON date?How can I pretty-print JSON in a shell script?What is JSON and why would I use it?How can I deserialize JSON to a simple Dictionary<string,string> in ASP.NET?How do I return clean JSON from a WCF Service?How can I parse a JSON file with PHP?How can I pretty-print JSON using JavaScript?How do I POST JSON data with Curl from a terminal/commandline to Test Spring REST?Post JSON using Python RequestsThe “right” JSON date format
Why would a propellor have blades of different lengths?
Misaligned Columns
I had an c.p.a file late returns, stating i would get money. but i.r.s. says they were filed too late
Can I deep fry food in butter instead of vegetable oil?
How did שְׁלֹמֹה (shlomo) become Solomon?
Bypass with wrong cvv of debit card and getting OTP
Olive oil in Japanese cooking
Phrase origin: "You ain't got to go home but you got to get out of here."
Do the 26 richest billionaires own as much wealth as the poorest 3.8 billion people?
Can I have a forest in the rain shadow of a mountain range?
Who pays for increased security measures on flights to the US?
Performance of loop vs expansion
Did Stalin kill all Soviet officers involved in the Winter War?
Find the closest HTML colour name
Is it possible that Curiosity measured its own methane or failed doing the spectrometry?
What are the differences of checking a self-signed certificate vs ignore it?
Does this circuit have marginal voltage level problem?
Which high-degree derivatives play an essential role?
Show that there are infinitely more problems than we will ever be able to compute
Where is read command?
Are the plates of a battery really charged?
What do you call the angle of the direction of an airplane?
What is meaning of 4 letter acronyms in Roman names like Titus Flavius T. f. T. n. Sabinus?
Phrasing "it says" or "it reads"
How to save a json from dataframe with specific format?
How do I format a Microsoft JSON date?How can I pretty-print JSON in a shell script?What is JSON and why would I use it?How can I deserialize JSON to a simple Dictionary<string,string> in ASP.NET?How do I return clean JSON from a WCF Service?How can I parse a JSON file with PHP?How can I pretty-print JSON using JavaScript?How do I POST JSON data with Curl from a terminal/commandline to Test Spring REST?Post JSON using Python RequestsThe “right” JSON date format
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
i'm trying to save the following pyspark dataframe to json:
ID VARIABLE_1
1 1
2 0
3 0
4 1
5 0
but i need the json to be like:
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
When saving to json i get the following:
"ID":"1", "VARIABLE_1": 1
"ID":"2", "VARIABLE_1": 0
"ID":"3", "VARIABLE_1": 0
"ID":"4", "VARIABLE_1": 1
"ID":"5", "VARIABLE_1": 0
i'm trying to save with
df.write.mode("overwrite").format("json").save(json_path)
i tried using create_map and got a dataframe like this:
NEW_COLUMN_NAME
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
But when i try to save this new dataframe to json i get:
"NEW_COLUMN_NAME":"1":"VARIABLE_1":1
"NEW_COLUMN_NAME":"2":"VARIABLE_1":0
"NEW_COLUMN_NAME":"3":"VARIABLE_1":0
"NEW_COLUMN_NAME":"4":"VARIABLE_1":1
"NEW_COLUMN_NAME":"5":"VARIABLE_1":0
I don't really know what to do, someone have any idea?
Thanks!
json pyspark
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
add a comment |
i'm trying to save the following pyspark dataframe to json:
ID VARIABLE_1
1 1
2 0
3 0
4 1
5 0
but i need the json to be like:
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
When saving to json i get the following:
"ID":"1", "VARIABLE_1": 1
"ID":"2", "VARIABLE_1": 0
"ID":"3", "VARIABLE_1": 0
"ID":"4", "VARIABLE_1": 1
"ID":"5", "VARIABLE_1": 0
i'm trying to save with
df.write.mode("overwrite").format("json").save(json_path)
i tried using create_map and got a dataframe like this:
NEW_COLUMN_NAME
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
But when i try to save this new dataframe to json i get:
"NEW_COLUMN_NAME":"1":"VARIABLE_1":1
"NEW_COLUMN_NAME":"2":"VARIABLE_1":0
"NEW_COLUMN_NAME":"3":"VARIABLE_1":0
"NEW_COLUMN_NAME":"4":"VARIABLE_1":1
"NEW_COLUMN_NAME":"5":"VARIABLE_1":0
I don't really know what to do, someone have any idea?
Thanks!
json pyspark
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
json format will contain the name and values, its not possible to drop one name (ID) and retain other name (VARIABLE_1) while saving as json format
– Ranga Vure
Mar 25 at 18:55
Once you get your data in the format you wanted (usingcreate_map) try using.write.text(json_path)to write the string to a file. Something like this:df.select("NEW_COLUMN_NAME").write.text(json_path)
– pault
Mar 25 at 19:12
when i try to write.text it says "Text data source does not support map" and if i try to Cast('string') it changes from "1": "VARIABLE_1": 1 to [1-> [VARIABLE_1-> 0]]
– TNS
Mar 25 at 19:59
My spark is version 2.4
– TNS
Mar 25 at 20:24
I'm not able to test on 2.4 but doesto_jsonwork on theMapType()column?df.select(to_json("NEW_COLUMN_NAME")).write.text(json_path). If not, it's a little hacky but you can build the string manually:df.select(concat(lit('"'), "ID", lit('":"VARIABLE_1":"'), "VARIABLE_1", lit('"')).write.text(json_path)
– pault
Mar 25 at 20:56
add a comment |
i'm trying to save the following pyspark dataframe to json:
ID VARIABLE_1
1 1
2 0
3 0
4 1
5 0
but i need the json to be like:
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
When saving to json i get the following:
"ID":"1", "VARIABLE_1": 1
"ID":"2", "VARIABLE_1": 0
"ID":"3", "VARIABLE_1": 0
"ID":"4", "VARIABLE_1": 1
"ID":"5", "VARIABLE_1": 0
i'm trying to save with
df.write.mode("overwrite").format("json").save(json_path)
i tried using create_map and got a dataframe like this:
NEW_COLUMN_NAME
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
But when i try to save this new dataframe to json i get:
"NEW_COLUMN_NAME":"1":"VARIABLE_1":1
"NEW_COLUMN_NAME":"2":"VARIABLE_1":0
"NEW_COLUMN_NAME":"3":"VARIABLE_1":0
"NEW_COLUMN_NAME":"4":"VARIABLE_1":1
"NEW_COLUMN_NAME":"5":"VARIABLE_1":0
I don't really know what to do, someone have any idea?
Thanks!
json pyspark
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
i'm trying to save the following pyspark dataframe to json:
ID VARIABLE_1
1 1
2 0
3 0
4 1
5 0
but i need the json to be like:
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
When saving to json i get the following:
"ID":"1", "VARIABLE_1": 1
"ID":"2", "VARIABLE_1": 0
"ID":"3", "VARIABLE_1": 0
"ID":"4", "VARIABLE_1": 1
"ID":"5", "VARIABLE_1": 0
i'm trying to save with
df.write.mode("overwrite").format("json").save(json_path)
i tried using create_map and got a dataframe like this:
NEW_COLUMN_NAME
"1": "VARIABLE_1": 1
"2": "VARIABLE_1": 0
"3": "VARIABLE_1": 0
"4": "VARIABLE_1": 1
"5": "VARIABLE_1": 0
But when i try to save this new dataframe to json i get:
"NEW_COLUMN_NAME":"1":"VARIABLE_1":1
"NEW_COLUMN_NAME":"2":"VARIABLE_1":0
"NEW_COLUMN_NAME":"3":"VARIABLE_1":0
"NEW_COLUMN_NAME":"4":"VARIABLE_1":1
"NEW_COLUMN_NAME":"5":"VARIABLE_1":0
I don't really know what to do, someone have any idea?
Thanks!
json pyspark
json pyspark
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
asked Mar 25 at 18:21
TNSTNS
112 bronze badges
112 bronze badges
json format will contain the name and values, its not possible to drop one name (ID) and retain other name (VARIABLE_1) while saving as json format
– Ranga Vure
Mar 25 at 18:55
Once you get your data in the format you wanted (usingcreate_map) try using.write.text(json_path)to write the string to a file. Something like this:df.select("NEW_COLUMN_NAME").write.text(json_path)
– pault
Mar 25 at 19:12
when i try to write.text it says "Text data source does not support map" and if i try to Cast('string') it changes from "1": "VARIABLE_1": 1 to [1-> [VARIABLE_1-> 0]]
– TNS
Mar 25 at 19:59
My spark is version 2.4
– TNS
Mar 25 at 20:24
I'm not able to test on 2.4 but doesto_jsonwork on theMapType()column?df.select(to_json("NEW_COLUMN_NAME")).write.text(json_path). If not, it's a little hacky but you can build the string manually:df.select(concat(lit('"'), "ID", lit('":"VARIABLE_1":"'), "VARIABLE_1", lit('"')).write.text(json_path)
– pault
Mar 25 at 20:56
add a comment |
json format will contain the name and values, its not possible to drop one name (ID) and retain other name (VARIABLE_1) while saving as json format
– Ranga Vure
Mar 25 at 18:55
Once you get your data in the format you wanted (usingcreate_map) try using.write.text(json_path)to write the string to a file. Something like this:df.select("NEW_COLUMN_NAME").write.text(json_path)
– pault
Mar 25 at 19:12
when i try to write.text it says "Text data source does not support map" and if i try to Cast('string') it changes from "1": "VARIABLE_1": 1 to [1-> [VARIABLE_1-> 0]]
– TNS
Mar 25 at 19:59
My spark is version 2.4
– TNS
Mar 25 at 20:24
I'm not able to test on 2.4 but doesto_jsonwork on theMapType()column?df.select(to_json("NEW_COLUMN_NAME")).write.text(json_path). If not, it's a little hacky but you can build the string manually:df.select(concat(lit('"'), "ID", lit('":"VARIABLE_1":"'), "VARIABLE_1", lit('"')).write.text(json_path)
– pault
Mar 25 at 20:56
json format will contain the name and values, its not possible to drop one name (ID) and retain other name (VARIABLE_1) while saving as json format
– Ranga Vure
Mar 25 at 18:55
json format will contain the name and values, its not possible to drop one name (ID) and retain other name (VARIABLE_1) while saving as json format
– Ranga Vure
Mar 25 at 18:55
Once you get your data in the format you wanted (using
create_map) try using .write.text(json_path) to write the string to a file. Something like this: df.select("NEW_COLUMN_NAME").write.text(json_path)– pault
Mar 25 at 19:12
Once you get your data in the format you wanted (using
create_map) try using .write.text(json_path) to write the string to a file. Something like this: df.select("NEW_COLUMN_NAME").write.text(json_path)– pault
Mar 25 at 19:12
when i try to write.text it says "Text data source does not support map" and if i try to Cast('string') it changes from "1": "VARIABLE_1": 1 to [1-> [VARIABLE_1-> 0]]
– TNS
Mar 25 at 19:59
when i try to write.text it says "Text data source does not support map" and if i try to Cast('string') it changes from "1": "VARIABLE_1": 1 to [1-> [VARIABLE_1-> 0]]
– TNS
Mar 25 at 19:59
My spark is version 2.4
– TNS
Mar 25 at 20:24
My spark is version 2.4
– TNS
Mar 25 at 20:24
I'm not able to test on 2.4 but does
to_json work on the MapType() column? df.select(to_json("NEW_COLUMN_NAME")).write.text(json_path). If not, it's a little hacky but you can build the string manually: df.select(concat(lit('"'), "ID", lit('":"VARIABLE_1":"'), "VARIABLE_1", lit('"')).write.text(json_path)– pault
Mar 25 at 20:56
I'm not able to test on 2.4 but does
to_json work on the MapType() column? df.select(to_json("NEW_COLUMN_NAME")).write.text(json_path). If not, it's a little hacky but you can build the string manually: df.select(concat(lit('"'), "ID", lit('":"VARIABLE_1":"'), "VARIABLE_1", lit('"')).write.text(json_path)– pault
Mar 25 at 20:56
add a comment |
1 Answer
1
active
oldest
votes
How i fixed the issue:
-Used collect() to collect the create_map dataframe
-Used for to make a dictionary with every row of the dataframe
-Used json.dumps with the dictionary
-Saved using open(path, "w")
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55344249%2fhow-to-save-a-json-from-dataframe-with-specific-format%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
How i fixed the issue:
-Used collect() to collect the create_map dataframe
-Used for to make a dictionary with every row of the dataframe
-Used json.dumps with the dictionary
-Saved using open(path, "w")
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
add a comment |
How i fixed the issue:
-Used collect() to collect the create_map dataframe
-Used for to make a dictionary with every row of the dataframe
-Used json.dumps with the dictionary
-Saved using open(path, "w")
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
add a comment |
How i fixed the issue:
-Used collect() to collect the create_map dataframe
-Used for to make a dictionary with every row of the dataframe
-Used json.dumps with the dictionary
-Saved using open(path, "w")
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
How i fixed the issue:
-Used collect() to collect the create_map dataframe
-Used for to make a dictionary with every row of the dataframe
-Used json.dumps with the dictionary
-Saved using open(path, "w")
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
answered Mar 27 at 18:56
TNSTNS
112 bronze badges
112 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55344249%2fhow-to-save-a-json-from-dataframe-with-specific-format%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
json format will contain the name and values, its not possible to drop one name (ID) and retain other name (VARIABLE_1) while saving as json format
– Ranga Vure
Mar 25 at 18:55
Once you get your data in the format you wanted (using
create_map) try using.write.text(json_path)to write the string to a file. Something like this:df.select("NEW_COLUMN_NAME").write.text(json_path)– pault
Mar 25 at 19:12
when i try to write.text it says "Text data source does not support map" and if i try to Cast('string') it changes from "1": "VARIABLE_1": 1 to [1-> [VARIABLE_1-> 0]]
– TNS
Mar 25 at 19:59
My spark is version 2.4
– TNS
Mar 25 at 20:24
I'm not able to test on 2.4 but does
to_jsonwork on theMapType()column?df.select(to_json("NEW_COLUMN_NAME")).write.text(json_path). If not, it's a little hacky but you can build the string manually:df.select(concat(lit('"'), "ID", lit('":"VARIABLE_1":"'), "VARIABLE_1", lit('"')).write.text(json_path)– pault
Mar 25 at 20:56