Styjun

Why would guns not work in the dungeon?

When did the Roman Empire fall according to contemporaries?

Would letting a multiclass character rebuild their character to be single-classed be game-breaking?

What is temperature on a quantum level?

TikZ Can I draw an arrow by specifying the initial point, direction, and length?

Help with understanding nuances of extremely popular Kyoto-ben (?) tweet

Was the Ford Model T black because of the speed black paint dries?

What are some symbols representing peasants/oppressed persons fighting back?

Why do they not say "The Baby"

Cubic programming and beyond?

Should you avoid redundant information after dialogue?

Find values of x so that the matrix is invertible

Why is dry soil hydrophobic? Bad gardener paradox

To accent or not to accent in Greek

P-MOSFET failing

Is this floating-point optimization allowed?

Chaining Dissonant Whispers via War Caster feat

Measuring mystery distances

Why does Hellboy file down his horns?

Draw 3D Cubes around centre

What to put after taking off rear stabilisers from child bicyle?

Supporting developers who insist on using their pet language

Interpreting the word "randomly"

How do Windows version numbers work?

How to rename column inside struct in spark scala [duplicate]

Rename nested field in spark dataframeAdding new Columns based on aggregation on existing column in Spark DataFrame using scalaSpark reading the Cassandra UDT columnDataframe column with two different namesSplitting a row in a PySpark Dataframe into multiple rowsHow to Pivot on Multiple column on Spark DataframeHow to pass a group of RelationalGroupedDataset to a function?Return two columns when mapping through a column list Spark SQL ScalaSpark/scala - can we create new columns from an existing column value in a dataframeSelecting a column not in cube in SparkHow to transpose row to column in spark/scala?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

This question already has an answer here:

• 
  Rename nested field in spark dataframe
  
  1 answer
  
  

I have a data frame. Which is like this -

 |-- Col1 : string (nullable = true)
 |-- Col2 : string (nullable = true)
 |-- Col3 : struct (nullable = true)
 | |-- 513: long (nullable = true)
 | |-- 549: long (nullable = true)

by using-

df.select("Col1","Col2","Col3.*").show

+-----------+--------+------+------+
| Col1| Col1| 513| 549|
+-----------+--------+------+------+
| AAAAAAAAA | BBBBB | 39| 38|
+-----------+--------+------+------+

Now I want to rename it

 +-----------+--------+---------+--------+
 | Col1| Col1| Col3=513|Col3=549|
 +-----------+--------+---------+--------+
 | AAAAAAAAA | BBBBB | 39| 38|
 +-----------+--------+---------+--------+

Columns inside struct is dynamic. So I can't use withColumnRenamed

scala apache-spark apache-spark-sql

share|improve this question

asked Mar 26 at 5:43

lucylucy

36933 gold badges99 silver badges2424 bronze badges

marked as duplicate by eliasah apache-spark
Users with the  apache-spark badge can single-handedly close apache-spark questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Mar 26 at 15:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

add a comment | 

1

This question already has an answer here:

• 
  Rename nested field in spark dataframe
  
  1 answer
  
  

I have a data frame. Which is like this -

 |-- Col1 : string (nullable = true)
 |-- Col2 : string (nullable = true)
 |-- Col3 : struct (nullable = true)
 | |-- 513: long (nullable = true)
 | |-- 549: long (nullable = true)

by using-

df.select("Col1","Col2","Col3.*").show

+-----------+--------+------+------+
| Col1| Col1| 513| 549|
+-----------+--------+------+------+
| AAAAAAAAA | BBBBB | 39| 38|
+-----------+--------+------+------+

Now I want to rename it

 +-----------+--------+---------+--------+
 | Col1| Col1| Col3=513|Col3=549|
 +-----------+--------+---------+--------+
 | AAAAAAAAA | BBBBB | 39| 38|
 +-----------+--------+---------+--------+

Columns inside struct is dynamic. So I can't use withColumnRenamed

scala apache-spark apache-spark-sql

share|improve this question

asked Mar 26 at 5:43

lucylucy

36933 gold badges99 silver badges2424 bronze badges

marked as duplicate by eliasah apache-spark
Users with the  apache-spark badge can single-handedly close apache-spark questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Mar 26 at 15:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

add a comment | 

This question already has an answer here:

• 
  Rename nested field in spark dataframe
  
  1 answer
  
  

I have a data frame. Which is like this -

 |-- Col1 : string (nullable = true)
 |-- Col2 : string (nullable = true)
 |-- Col3 : struct (nullable = true)
 | |-- 513: long (nullable = true)
 | |-- 549: long (nullable = true)

by using-

df.select("Col1","Col2","Col3.*").show

+-----------+--------+------+------+
| Col1| Col1| 513| 549|
+-----------+--------+------+------+
| AAAAAAAAA | BBBBB | 39| 38|
+-----------+--------+------+------+

Now I want to rename it

 +-----------+--------+---------+--------+
 | Col1| Col1| Col3=513|Col3=549|
 +-----------+--------+---------+--------+
 | AAAAAAAAA | BBBBB | 39| 38|
 +-----------+--------+---------+--------+

Columns inside struct is dynamic. So I can't use withColumnRenamed

scala apache-spark apache-spark-sql

share|improve this question

asked Mar 26 at 5:43

lucylucy

36933 gold badges99 silver badges2424 bronze badges

 |-- Col1 : string (nullable = true)
 |-- Col2 : string (nullable = true)
 |-- Col3 : struct (nullable = true)
 | |-- 513: long (nullable = true)
 | |-- 549: long (nullable = true)

df.select("Col1","Col2","Col3.*").show

+-----------+--------+------+------+
| Col1| Col1| 513| 549|
+-----------+--------+------+------+
| AAAAAAAAA | BBBBB | 39| 38|
+-----------+--------+------+------+

 +-----------+--------+---------+--------+
 | Col1| Col1| Col3=513|Col3=549|
 +-----------+--------+---------+--------+
 | AAAAAAAAA | BBBBB | 39| 38|
 +-----------+--------+---------+--------+

share|improve this question

asked Mar 26 at 5:43

lucylucy

36933 gold badges99 silver badges2424 bronze badges

share|improve this question

asked Mar 26 at 5:43

lucylucy

36933 gold badges99 silver badges2424 bronze badges

asked Mar 26 at 5:43

lucylucy

36933 gold badges99 silver badges2424 bronze badges

asked Mar 26 at 5:43

lucylucy

36933 gold badges99 silver badges2424 bronze badges

1 Answer
1

active

oldest

votes

0

As you ask about renaming insude structs, you can achieve this using Schema DSL:

import org.apache.spark.sql.types._

val schema: StructType = df.schema.fields.find(_.name=="Col3").get.dataType.asInstanceOf[StructType]
val newSchema = StructType.apply(schema.fields.map(sf => StructField.apply("Col3="+sf.name,sf.dataType)))

df
 .withColumn("Col3",$"Col3".cast(newSchema))
 .printSchema()

gives

root
 |-- Col1: string (nullable = true)
 |-- Col2: string (nullable = true)
 |-- Col3: struct (nullable = false)
 | |-- Col3=513: long (nullable = true)
 | |-- Col3=549: long (nullable = true)

Then you can unpack it using select($"col3.*").

You could also unpack the struct first and then rename all the columns which have an number as column name...

share|improve this answer

answered Mar 26 at 6:20

Raphael RothRaphael Roth

13.5k55 gold badges4646 silver badges8585 bronze badges

add a comment | 

0

As you ask about renaming insude structs, you can achieve this using Schema DSL:

import org.apache.spark.sql.types._

val schema: StructType = df.schema.fields.find(_.name=="Col3").get.dataType.asInstanceOf[StructType]
val newSchema = StructType.apply(schema.fields.map(sf => StructField.apply("Col3="+sf.name,sf.dataType)))

df
 .withColumn("Col3",$"Col3".cast(newSchema))
 .printSchema()

gives

root
 |-- Col1: string (nullable = true)
 |-- Col2: string (nullable = true)
 |-- Col3: struct (nullable = false)
 | |-- Col3=513: long (nullable = true)
 | |-- Col3=549: long (nullable = true)

Then you can unpack it using select($"col3.*").

You could also unpack the struct first and then rename all the columns which have an number as column name...

share|improve this answer

answered Mar 26 at 6:20

Raphael RothRaphael Roth

13.5k55 gold badges4646 silver badges8585 bronze badges

add a comment | 

0

As you ask about renaming insude structs, you can achieve this using Schema DSL:

import org.apache.spark.sql.types._

val schema: StructType = df.schema.fields.find(_.name=="Col3").get.dataType.asInstanceOf[StructType]
val newSchema = StructType.apply(schema.fields.map(sf => StructField.apply("Col3="+sf.name,sf.dataType)))

df
 .withColumn("Col3",$"Col3".cast(newSchema))
 .printSchema()

gives

root
 |-- Col1: string (nullable = true)
 |-- Col2: string (nullable = true)
 |-- Col3: struct (nullable = false)
 | |-- Col3=513: long (nullable = true)
 | |-- Col3=549: long (nullable = true)

Then you can unpack it using select($"col3.*").

You could also unpack the struct first and then rename all the columns which have an number as column name...

share|improve this answer

answered Mar 26 at 6:20

Raphael RothRaphael Roth

13.5k55 gold badges4646 silver badges8585 bronze badges

add a comment | 

As you ask about renaming insude structs, you can achieve this using Schema DSL:

import org.apache.spark.sql.types._

val schema: StructType = df.schema.fields.find(_.name=="Col3").get.dataType.asInstanceOf[StructType]
val newSchema = StructType.apply(schema.fields.map(sf => StructField.apply("Col3="+sf.name,sf.dataType)))

df
 .withColumn("Col3",$"Col3".cast(newSchema))
 .printSchema()

gives

root
 |-- Col1: string (nullable = true)
 |-- Col2: string (nullable = true)
 |-- Col3: struct (nullable = false)
 | |-- Col3=513: long (nullable = true)
 | |-- Col3=549: long (nullable = true)

Then you can unpack it using select($"col3.*").

You could also unpack the struct first and then rename all the columns which have an number as column name...

share|improve this answer

answered Mar 26 at 6:20

Raphael RothRaphael Roth

13.5k55 gold badges4646 silver badges8585 bronze badges

import org.apache.spark.sql.types._

val schema: StructType = df.schema.fields.find(_.name=="Col3").get.dataType.asInstanceOf[StructType]
val newSchema = StructType.apply(schema.fields.map(sf => StructField.apply("Col3="+sf.name,sf.dataType)))

df
 .withColumn("Col3",$"Col3".cast(newSchema))
 .printSchema()

root
 |-- Col1: string (nullable = true)
 |-- Col2: string (nullable = true)
 |-- Col3: struct (nullable = false)
 | |-- Col3=513: long (nullable = true)
 | |-- Col3=549: long (nullable = true)

share|improve this answer

answered Mar 26 at 6:20

Raphael RothRaphael Roth

13.5k55 gold badges4646 silver badges8585 bronze badges

answered Mar 26 at 6:20

Raphael RothRaphael Roth

13.5k55 gold badges4646 silver badges8585 bronze badges

answered Mar 26 at 6:20

Raphael RothRaphael Roth

13.5k55 gold badges4646 silver badges8585 bronze badges