Styjun

Is this cheap "air conditioner" able to cool a room?

Did silent film actors actually say their lines or did they simply improvise “dialogue” while being filmed?

one fifteen euros=115 euros: Is there a german equivalent?

Our group keeps dying during the Lost Mine of Phandelver campaign. What are we doing wrong?

Is it a bad idea to offer variants of a final exam based on the type of allowed calculators?

Do any languages mention the top limit of a range first?

sytemctl status log output

In the movie Harry Potter and the Order or the Phoenix, why didn't Mr. Filch succeed to open the Room of Requirement if it's what he needed?

Why do proponents of guns oppose gun competency tests?

What is a Dominant Word™?

Why are the inside diameters of some pipe larger than the stated size?

Looking for a new job because of relocation - is it okay to tell the real reason?

Traveling from Germany to other countries by train?

Would the Elder Wand have been able to destroy a Horcrux?

Secure my password from unsafe servers

Decode a variable-length quantity

Did WWII Japanese soldiers engage in cannibalism of their enemies?

What are these silver stripes on Cosmic Girl for?

What does VB stand for?

Unexpected route on a flight from USA to Europe

Does this put me at risk for identity theft?

Is there a loss of quality when converting RGB to HEX?

How many years before enough atoms of your body are replaced to survive the sudden disappearance of the original body’s atoms?

Short story about a teenager who has his brain replaced with a microchip (Psychological Horror)

spark 1.6 - RDD creation and history

Difference between DataFrame, Dataset, and RDD in SparkHow can one see what RDD is available to Spark?Spark streaming in python: bugs in countByValue and countByValueAndWindow?Spark streaming persistent RDD partitions separationPySpark RDD filtered-out elements coming backSpark 2.0: Redefining SparkSession params through GetOrCreate and NOT seeing changes in WebUIHow Python data structure implemented in Spark when using PySpark?Converting Python Dict to Sparse RDD or DF in PySparkHandling missing data in PysparkHow do I transform dataframe column with unicode data in PySpark

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

I am using pyspark 1.6 with Python and I would like to keep track of the list of previous sessions and the history .

For example : RDD created 2 days back with persist option .
created RDD and it was active for the current session only and when I exit the session I dont see the track of the previous transformations .

Any settings or parameter to be configured ?

Thanks
Vijay

apache-spark pyspark

share|improve this question

edited Mar 27 at 8:19

howie

1,62922 gold badges1414 silver badges2424 bronze badges

asked Mar 27 at 5:50

Vee JayBeeVee JayBee

333 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  What do you mean sessions ? SparkSession ?
  
  – howie
  Mar 27 at 6:03
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  yes , the spark session that gets started when we hit -pyspark command
  
  – Vee JayBee
  Mar 27 at 6:13
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The session only live in runtime. So I guest you want to see history log form Spark UI?
  
  – howie
  Mar 27 at 6:17
  

  
  
  
  

add a comment | 

0

I am using pyspark 1.6 with Python and I would like to keep track of the list of previous sessions and the history .

For example : RDD created 2 days back with persist option .
created RDD and it was active for the current session only and when I exit the session I dont see the track of the previous transformations .

Any settings or parameter to be configured ?

Thanks
Vijay

apache-spark pyspark

share|improve this question

edited Mar 27 at 8:19

howie

1,62922 gold badges1414 silver badges2424 bronze badges

asked Mar 27 at 5:50

Vee JayBeeVee JayBee

333 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  What do you mean sessions ? SparkSession ?
  
  – howie
  Mar 27 at 6:03
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  yes , the spark session that gets started when we hit -pyspark command
  
  – Vee JayBee
  Mar 27 at 6:13
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  The session only live in runtime. So I guest you want to see history log form Spark UI?
  
  – howie
  Mar 27 at 6:17
  

  
  
  
  

add a comment | 

I am using pyspark 1.6 with Python and I would like to keep track of the list of previous sessions and the history .

For example : RDD created 2 days back with persist option .
created RDD and it was active for the current session only and when I exit the session I dont see the track of the previous transformations .

Any settings or parameter to be configured ?

Thanks
Vijay

apache-spark pyspark

share|improve this question

edited Mar 27 at 8:19

howie

1,62922 gold badges1414 silver badges2424 bronze badges

asked Mar 27 at 5:50

Vee JayBeeVee JayBee

333 bronze badges

share|improve this question

edited Mar 27 at 8:19

howie

1,62922 gold badges1414 silver badges2424 bronze badges

asked Mar 27 at 5:50

Vee JayBeeVee JayBee

333 bronze badges

share|improve this question

edited Mar 27 at 8:19

howie

1,62922 gold badges1414 silver badges2424 bronze badges

asked Mar 27 at 5:50

Vee JayBeeVee JayBee

333 bronze badges

edited Mar 27 at 8:19

howie

1,62922 gold badges1414 silver badges2424 bronze badges

edited Mar 27 at 8:19

howie

1,62922 gold badges1414 silver badges2424 bronze badges

asked Mar 27 at 5:50

Vee JayBeeVee JayBee

333 bronze badges

asked Mar 27 at 5:50

Vee JayBeeVee JayBee

333 bronze badges

1 Answer
1

active

oldest

votes

0

You can use :history command in your spark REPL.
It will provide all the commands used for current session.

scala> :history
 1 exit
 2 quit
 3 quit;
 4 close();
 5 val rdd = sc.textFile(“README.md”)
 6 val rdd = sc.textFile("README.md")
 7 rdd.count();
 8 val rdd = sc.textFile("README.md")
 9 rdd.count()
 10 val rdd = sc.textFile("/Users/**/bashrc.txt")
 11 rdd.count()
 12 val rdd = sc.textFile("README.md")
 13 rdd.count();
 14 val rdd = sc.textFile("README.md")
 15 rdd.count()
 16 quit
 17 exit
 18 exit;
 19 history
 20 :history


 scala> 

share|improve this answer

answered Mar 27 at 12:32

KayVKayV

4,71244 gold badges3131 silver badges7272 bronze badges

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55370601%2fspark-1-6-rdd-creation-and-history%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

You can use :history command in your spark REPL.
It will provide all the commands used for current session.

scala> :history
 1 exit
 2 quit
 3 quit;
 4 close();
 5 val rdd = sc.textFile(“README.md”)
 6 val rdd = sc.textFile("README.md")
 7 rdd.count();
 8 val rdd = sc.textFile("README.md")
 9 rdd.count()
 10 val rdd = sc.textFile("/Users/**/bashrc.txt")
 11 rdd.count()
 12 val rdd = sc.textFile("README.md")
 13 rdd.count();
 14 val rdd = sc.textFile("README.md")
 15 rdd.count()
 16 quit
 17 exit
 18 exit;
 19 history
 20 :history


 scala> 

share|improve this answer

answered Mar 27 at 12:32

KayVKayV

4,71244 gold badges3131 silver badges7272 bronze badges

add a comment | 

0

You can use :history command in your spark REPL.
It will provide all the commands used for current session.

scala> :history
 1 exit
 2 quit
 3 quit;
 4 close();
 5 val rdd = sc.textFile(“README.md”)
 6 val rdd = sc.textFile("README.md")
 7 rdd.count();
 8 val rdd = sc.textFile("README.md")
 9 rdd.count()
 10 val rdd = sc.textFile("/Users/**/bashrc.txt")
 11 rdd.count()
 12 val rdd = sc.textFile("README.md")
 13 rdd.count();
 14 val rdd = sc.textFile("README.md")
 15 rdd.count()
 16 quit
 17 exit
 18 exit;
 19 history
 20 :history


 scala> 

share|improve this answer

answered Mar 27 at 12:32

KayVKayV

4,71244 gold badges3131 silver badges7272 bronze badges

add a comment | 

You can use :history command in your spark REPL.
It will provide all the commands used for current session.

scala> :history
 1 exit
 2 quit
 3 quit;
 4 close();
 5 val rdd = sc.textFile(“README.md”)
 6 val rdd = sc.textFile("README.md")
 7 rdd.count();
 8 val rdd = sc.textFile("README.md")
 9 rdd.count()
 10 val rdd = sc.textFile("/Users/**/bashrc.txt")
 11 rdd.count()
 12 val rdd = sc.textFile("README.md")
 13 rdd.count();
 14 val rdd = sc.textFile("README.md")
 15 rdd.count()
 16 quit
 17 exit
 18 exit;
 19 history
 20 :history


 scala> 

share|improve this answer

answered Mar 27 at 12:32

KayVKayV

4,71244 gold badges3131 silver badges7272 bronze badges

scala> :history
 1 exit
 2 quit
 3 quit;
 4 close();
 5 val rdd = sc.textFile(“README.md”)
 6 val rdd = sc.textFile("README.md")
 7 rdd.count();
 8 val rdd = sc.textFile("README.md")
 9 rdd.count()
 10 val rdd = sc.textFile("/Users/**/bashrc.txt")
 11 rdd.count()
 12 val rdd = sc.textFile("README.md")
 13 rdd.count();
 14 val rdd = sc.textFile("README.md")
 15 rdd.count()
 16 quit
 17 exit
 18 exit;
 19 history
 20 :history


 scala> 

share|improve this answer

answered Mar 27 at 12:32

KayVKayV

4,71244 gold badges3131 silver badges7272 bronze badges

answered Mar 27 at 12:32

KayVKayV

4,71244 gold badges3131 silver badges7272 bronze badges

answered Mar 27 at 12:32

KayVKayV

4,71244 gold badges3131 silver badges7272 bronze badges