specify marksweep gc for `spark-submit` job on emrHow to pass -D parameter or environment variable to Spark job?File not found exception when submitting spark job to EMRwhat to specify as spark master when running on amazon emrSpark jobs are not running on emrSpark Job Submission: AWS EMR step or command line spark-submitHow to execute spark submit on amazon EMR from Lambda function?ClassNotFoundException while spark-submitting a JAR on EMRhive on spark on amazon EMREMR Spark Job computes less results each time it's runSubmitting Spark job to Amazon EMR
How can I use my cell phone's light as a reading light?
How was the website able to tell my credit card was wrong before it processed it?
Forward DNS request to my work's jump server
Four ships at the ocean with the same distance
The Apéry's constant and the Airy function
Can one block with a protection from color creature?
What are the effects of abstaining from eating a certain flavor?
Is there a formal/better word than "skyrocket" for the given context?
Curly braces adjustment in tikz?
Where are the Wazirs?
Is space division multiplexing really multiplexing?
Interpretation of non-significant results as "trends"
is there a real life meaning about KMeans error?
How do I explain that I don't want to maintain old projects?
Matrices with shadows
Category-theoretic treatment of diffs, patches and merging?
What does the multimeter dial do internally?
Why is a semiprime used as the modulus in RSA?
Passwordless authentication - how and when to invalidate a login code
What is the meaning of "prairie-dog" in this sentence?
Computer name naming convention for security
Intern not wearing safety equipment; how could I have handled this differently?
Stop Gravel from Washing Away
QR codes, do people use them?
specify marksweep gc for `spark-submit` job on emr
How to pass -D parameter or environment variable to Spark job?File not found exception when submitting spark job to EMRwhat to specify as spark master when running on amazon emrSpark jobs are not running on emrSpark Job Submission: AWS EMR step or command line spark-submitHow to execute spark submit on amazon EMR from Lambda function?ClassNotFoundException while spark-submitting a JAR on EMRhive on spark on amazon EMREMR Spark Job computes less results each time it's runSubmitting Spark job to Amazon EMR
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
How do I specify that I want the jvm to use the MarkSweep gc when running a spark-submit job on emr? Can I submit it with the job (i.e. spark-submit -- conf...) and if so, what is the command? Does this have to be set by the time spark starts and, if so, how do I specify that in the emr config?
thanks!
apache-spark garbage-collection jvm amazon-emr
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
add a comment |
How do I specify that I want the jvm to use the MarkSweep gc when running a spark-submit job on emr? Can I submit it with the job (i.e. spark-submit -- conf...) and if so, what is the command? Does this have to be set by the time spark starts and, if so, how do I specify that in the emr config?
thanks!
apache-spark garbage-collection jvm amazon-emr
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
add a comment |
How do I specify that I want the jvm to use the MarkSweep gc when running a spark-submit job on emr? Can I submit it with the job (i.e. spark-submit -- conf...) and if so, what is the command? Does this have to be set by the time spark starts and, if so, how do I specify that in the emr config?
thanks!
apache-spark garbage-collection jvm amazon-emr
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
How do I specify that I want the jvm to use the MarkSweep gc when running a spark-submit job on emr? Can I submit it with the job (i.e. spark-submit -- conf...) and if so, what is the command? Does this have to be set by the time spark starts and, if so, how do I specify that in the emr config?
thanks!
apache-spark garbage-collection jvm amazon-emr
apache-spark garbage-collection jvm amazon-emr
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
asked Mar 25 at 22:15
Walrus the CatWalrus the Cat
7872 gold badges18 silver badges48 bronze badges
7872 gold badges18 silver badges48 bronze badges
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You can specify JVM options in the --conf "spark.driver.extraJavaOptions=<YourJavaOptionsHere>" following the spark-submit.
For example, running PageRank example of graphX with ParallelScavenge GC would be like this:
spark-submit run-example --conf "spark.driver.extraJavaOptions=-XX:+UseParallelGC" graphx.PageRankExample
More information is at https://spark.apache.org/docs/latest/configuration.html#runtime-environment
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55347193%2fspecify-marksweep-gc-for-spark-submit-job-on-emr%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can specify JVM options in the --conf "spark.driver.extraJavaOptions=<YourJavaOptionsHere>" following the spark-submit.
For example, running PageRank example of graphX with ParallelScavenge GC would be like this:
spark-submit run-example --conf "spark.driver.extraJavaOptions=-XX:+UseParallelGC" graphx.PageRankExample
More information is at https://spark.apache.org/docs/latest/configuration.html#runtime-environment
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
add a comment |
You can specify JVM options in the --conf "spark.driver.extraJavaOptions=<YourJavaOptionsHere>" following the spark-submit.
For example, running PageRank example of graphX with ParallelScavenge GC would be like this:
spark-submit run-example --conf "spark.driver.extraJavaOptions=-XX:+UseParallelGC" graphx.PageRankExample
More information is at https://spark.apache.org/docs/latest/configuration.html#runtime-environment
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
add a comment |
You can specify JVM options in the --conf "spark.driver.extraJavaOptions=<YourJavaOptionsHere>" following the spark-submit.
For example, running PageRank example of graphX with ParallelScavenge GC would be like this:
spark-submit run-example --conf "spark.driver.extraJavaOptions=-XX:+UseParallelGC" graphx.PageRankExample
More information is at https://spark.apache.org/docs/latest/configuration.html#runtime-environment
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
You can specify JVM options in the --conf "spark.driver.extraJavaOptions=<YourJavaOptionsHere>" following the spark-submit.
For example, running PageRank example of graphX with ParallelScavenge GC would be like this:
spark-submit run-example --conf "spark.driver.extraJavaOptions=-XX:+UseParallelGC" graphx.PageRankExample
More information is at https://spark.apache.org/docs/latest/configuration.html#runtime-environment
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
answered Mar 26 at 1:17
HowieHowie
216 bronze badges
216 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55347193%2fspecify-marksweep-gc-for-spark-submit-job-on-emr%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
