Styjun

Is Tales of Old an official bard ability?

"This used to be my phone number"

Manager asking me to eat breakfast from now on

Why are there few or no black super GMs?

Does unblocking power bar outlets through short extension cords increase fire risk?

Will the internet speed decrease on second router if there are multiple devices connected to primary router?

How slow ( not zero) can a car engine run without hurting engine and saving on fuel

Changing iteration variable in Do loop

In this iconic lunar orbit rendezvous photo of John Houbolt, why do arrows #5 and #6 point the "wrong" way?

Is this Android phone Android 9.0 or Android 6.0?

literal `0` beeing a valid candidate for int and const string& overloads causes ambiguous call

Should I have shared a document with a former employee?

Why is Google approaching my VPS machine?

Operation Unzalgo

Who determines when road center lines are solid or dashed?

How to interpret a promising preprint that was never published in peer-review?

Did Hitler say this quote about homeschooling?

Why teach C using scanf without talking about command line arguments?

Do pedestrians imitate auto traffic?

Why did my "seldom" get corrected?

Company looks for long-term employees, but I know I won't be interested in staying long

How can I automate this tensor computation?

Real orthogonal and sign

Why is the Intel 8086 CPU called a 16-bit CPU?

Parse JSON stored in table column in Python Spark

How do I parse XML in Python?Why can't Python parse this JSON data?Adding new column to existing DataFrame in Python pandas“Large data” work flows using pandasExtract Nested Json fields from DynamoDB JSON string using Spark?Updating column in spark dataframe with json schemaExtracting values from a Spark column containing nested valueswhat is optimal way to parse following kafka JSON message to pyspark dataframe?Parsing CSV file with JSON array with ScalaPython - Parsing JSON Data through user defined function

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

0

I'm trying to parse JSON and add a column to a DataFrame using Python Spark:

tableDF = spark.sql("select * from transaction")

stats_df = parseJSONCols(tableDF)

def parseJSONCols(df):
 res = df
 cols = ['State']

 for i in cols:
 schema = spark.read
 .json(res.rdd.map(lambda x: x[i]))
 .schema
 res = res.withColumn("selectedState", lit(filterSelectedState(col(i))))

 return res

where State is a JSON string with this structure:

[
 
 isSelected: true,
 name: 'x'
 ,
 
 isSelected: false,
 name: 'y'
 
]

I want to extract the value of the isSelected field and add that in a selected column in tableDF.

Can someone please help me achieve this?

python pyspark databricks

share|improve this question

edited Jun 8 at 16:54

nclark

53911 gold badge44 silver badges1414 bronze badges

asked Mar 26 at 10:43

Arun TyagiArun Tyagi

87133 gold badges1313 silver badges2828 bronze badges

add a comment | 

0

0

I'm trying to parse JSON and add a column to a DataFrame using Python Spark:

tableDF = spark.sql("select * from transaction")

stats_df = parseJSONCols(tableDF)

def parseJSONCols(df):
 res = df
 cols = ['State']

 for i in cols:
 schema = spark.read
 .json(res.rdd.map(lambda x: x[i]))
 .schema
 res = res.withColumn("selectedState", lit(filterSelectedState(col(i))))

 return res

where State is a JSON string with this structure:

[
 
 isSelected: true,
 name: 'x'
 ,
 
 isSelected: false,
 name: 'y'
 
]

I want to extract the value of the isSelected field and add that in a selected column in tableDF.

Can someone please help me achieve this?

python pyspark databricks

share|improve this question

edited Jun 8 at 16:54

nclark

53911 gold badge44 silver badges1414 bronze badges

asked Mar 26 at 10:43

Arun TyagiArun Tyagi

87133 gold badges1313 silver badges2828 bronze badges

add a comment | 

I'm trying to parse JSON and add a column to a DataFrame using Python Spark:

tableDF = spark.sql("select * from transaction")

stats_df = parseJSONCols(tableDF)

def parseJSONCols(df):
 res = df
 cols = ['State']

 for i in cols:
 schema = spark.read
 .json(res.rdd.map(lambda x: x[i]))
 .schema
 res = res.withColumn("selectedState", lit(filterSelectedState(col(i))))

 return res

where State is a JSON string with this structure:

[
 
 isSelected: true,
 name: 'x'
 ,
 
 isSelected: false,
 name: 'y'
 
]

I want to extract the value of the isSelected field and add that in a selected column in tableDF.

Can someone please help me achieve this?

python pyspark databricks

share|improve this question

edited Jun 8 at 16:54

nclark

53911 gold badge44 silver badges1414 bronze badges

asked Mar 26 at 10:43

Arun TyagiArun Tyagi

87133 gold badges1313 silver badges2828 bronze badges

tableDF = spark.sql("select * from transaction")

stats_df = parseJSONCols(tableDF)

def parseJSONCols(df):
 res = df
 cols = ['State']

 for i in cols:
 schema = spark.read
 .json(res.rdd.map(lambda x: x[i]))
 .schema
 res = res.withColumn("selectedState", lit(filterSelectedState(col(i))))

 return res

[
 
 isSelected: true,
 name: 'x'
 ,
 
 isSelected: false,
 name: 'y'
 
]

share|improve this question

edited Jun 8 at 16:54

nclark

53911 gold badge44 silver badges1414 bronze badges

asked Mar 26 at 10:43

Arun TyagiArun Tyagi

87133 gold badges1313 silver badges2828 bronze badges

share|improve this question

edited Jun 8 at 16:54

nclark

53911 gold badge44 silver badges1414 bronze badges

asked Mar 26 at 10:43

Arun TyagiArun Tyagi

87133 gold badges1313 silver badges2828 bronze badges

edited Jun 8 at 16:54

nclark

53911 gold badge44 silver badges1414 bronze badges

edited Jun 8 at 16:54

nclark

53911 gold badge44 silver badges1414 bronze badges

asked Mar 26 at 10:43

Arun TyagiArun Tyagi

87133 gold badges1313 silver badges2828 bronze badges

asked Mar 26 at 10:43

Arun TyagiArun Tyagi

87133 gold badges1313 silver badges2828 bronze badges

1 Answer
1

active

oldest

votes

0

You probably want something like:

df = df.withColumn('isSelected', f.col('State').getItem('isSelected'))

However, it would be useful to see the output of say stats_df.take(2) so that we can see the exact structure of DF you are dealing with.

share|improve this answer

answered Mar 26 at 12:20

ags29ags29

1,14911 gold badge22 silver badges77 bronze badges

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55355191%2fparse-json-stored-in-table-column-in-python-spark%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

You probably want something like:

df = df.withColumn('isSelected', f.col('State').getItem('isSelected'))

However, it would be useful to see the output of say stats_df.take(2) so that we can see the exact structure of DF you are dealing with.

share|improve this answer

answered Mar 26 at 12:20

ags29ags29

1,14911 gold badge22 silver badges77 bronze badges

add a comment | 

0

You probably want something like:

df = df.withColumn('isSelected', f.col('State').getItem('isSelected'))

However, it would be useful to see the output of say stats_df.take(2) so that we can see the exact structure of DF you are dealing with.

share|improve this answer

answered Mar 26 at 12:20

ags29ags29

1,14911 gold badge22 silver badges77 bronze badges

add a comment | 

You probably want something like:

df = df.withColumn('isSelected', f.col('State').getItem('isSelected'))

However, it would be useful to see the output of say stats_df.take(2) so that we can see the exact structure of DF you are dealing with.

share|improve this answer

answered Mar 26 at 12:20

ags29ags29

1,14911 gold badge22 silver badges77 bronze badges

df = df.withColumn('isSelected', f.col('State').getItem('isSelected'))

share|improve this answer

answered Mar 26 at 12:20

ags29ags29

1,14911 gold badge22 silver badges77 bronze badges

answered Mar 26 at 12:20

ags29ags29

1,14911 gold badge22 silver badges77 bronze badges

answered Mar 26 at 12:20

ags29ags29

1,14911 gold badge22 silver badges77 bronze badges