Is it possible to extend the scheduler of Apache Spark?How to control preferred locations of RDD partitions?Apache Spark vs AkkaApache Spark: Applying a function from sklearn parallel on partitionsLambda Architecture with Apache SparkStorm like structure in Apache SparkApache Spark Multi Node Clustering - java.io.FileNotFoundExceptionHow are stages split into tasks in Spark?How to connect elasticsearch to apache spark streaming or storm?Stateful and Stateless Streaming (Spark)Schedule each Apache Spark Stage to run on a specific Worker NodeMerging Neo4j and HBase data using Apache spark
Company threw a surprise party for the CEO, 3 weeks later management says we have to pay for it, do I have to?
Intersecting with the x-axis / intersecting the x-axis
Why do Thanos' punches not kill Captain America or at least cause some mortal injuries?
Is there any evidence to support the claim that the United States was "suckered into WW1" by Zionists, made by Benjamin Freedman in his 1961 speech
How to evaluate sum with one million summands?
Company stopped paying my salary. What are my options?
Ex-manager wants to stay in touch, I don't want to
The meaning of a て-form verb at the end of this sentence
date to display the EDT time
Why do the Avengers care about returning these items in Endgame?
Does the 500 feet falling cap apply per fall, or per turn?
What does formal training in a field mean?
Is it nonsense to say B -> [A -> B]?
Cropping a message using array splits
Best species to breed to intelligence
No such column 'DeveloperName' on entity 'RecordType' after Summer '19 release on sandbox
Names of the Six Tastes
Was there a contingency plan in place if Little Boy failed to detonate?
Why was the ancient one so hesitant to teach Dr Strange the art of sorcery
How is CoreiX like Corei5, i7 is related to Haswell, Ivy Bridge?
Why is it wrong to *implement* myself a known, published, widely believed to be secure crypto algorithm?
Is this state of Earth possible, after humans left for a million years?
Is it a good idea to copy a trader when investing?
Thesis' "Future Work" section – is it acceptable to omit personal involvement in a mentioned project?
Is it possible to extend the scheduler of Apache Spark?
How to control preferred locations of RDD partitions?Apache Spark vs AkkaApache Spark: Applying a function from sklearn parallel on partitionsLambda Architecture with Apache SparkStorm like structure in Apache SparkApache Spark Multi Node Clustering - java.io.FileNotFoundExceptionHow are stages split into tasks in Spark?How to connect elasticsearch to apache spark streaming or storm?Stateful and Stateless Streaming (Spark)Schedule each Apache Spark Stage to run on a specific Worker NodeMerging Neo4j and HBase data using Apache spark
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I only know the basics of Apache Spark framework. Is it possible to extend Spark scheduler to place its task's operators on different node machines like it is possible in Apache Storm?
On Apache Storm is only implement the IScheduler and its methods.
apache-spark
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
add a comment |
I only know the basics of Apache Spark framework. Is it possible to extend Spark scheduler to place its task's operators on different node machines like it is possible in Apache Storm?
On Apache Storm is only implement the IScheduler and its methods.
apache-spark
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
1
If the question is if the API is intended to be extended by the end users, then the answer is most likely negative. And to quote Felix Cheung response to acceleration aware scheduling - Third, the proposal touches on some fairly core and sensitive components, like the scheduler, and I think more discussions are necessary. We have a few comments there and in the JIRA.. Have you rather looked at prefferedLocations?
– user10958683
Mar 23 at 10:22
Hi @user10958683, thanks for your answer. Seems Flink also have this requirement (docs.google.com/document/d/…). About "prefferedLocations", I don't know what it is in Spark. Where I can find some reference? I searched and couldn't find. thanks
– Felipe
Mar 23 at 10:47
1
Take a look at this answer by Jacek Laskowski.
– user10958683
Mar 23 at 11:44
add a comment |
I only know the basics of Apache Spark framework. Is it possible to extend Spark scheduler to place its task's operators on different node machines like it is possible in Apache Storm?
On Apache Storm is only implement the IScheduler and its methods.
apache-spark
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
I only know the basics of Apache Spark framework. Is it possible to extend Spark scheduler to place its task's operators on different node machines like it is possible in Apache Storm?
On Apache Storm is only implement the IScheduler and its methods.
apache-spark
apache-spark
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
asked Mar 23 at 10:02
FelipeFelipe
1,07721433
1,07721433
1
If the question is if the API is intended to be extended by the end users, then the answer is most likely negative. And to quote Felix Cheung response to acceleration aware scheduling - Third, the proposal touches on some fairly core and sensitive components, like the scheduler, and I think more discussions are necessary. We have a few comments there and in the JIRA.. Have you rather looked at prefferedLocations?
– user10958683
Mar 23 at 10:22
Hi @user10958683, thanks for your answer. Seems Flink also have this requirement (docs.google.com/document/d/…). About "prefferedLocations", I don't know what it is in Spark. Where I can find some reference? I searched and couldn't find. thanks
– Felipe
Mar 23 at 10:47
1
Take a look at this answer by Jacek Laskowski.
– user10958683
Mar 23 at 11:44
add a comment |
1
If the question is if the API is intended to be extended by the end users, then the answer is most likely negative. And to quote Felix Cheung response to acceleration aware scheduling - Third, the proposal touches on some fairly core and sensitive components, like the scheduler, and I think more discussions are necessary. We have a few comments there and in the JIRA.. Have you rather looked at prefferedLocations?
– user10958683
Mar 23 at 10:22
Hi @user10958683, thanks for your answer. Seems Flink also have this requirement (docs.google.com/document/d/…). About "prefferedLocations", I don't know what it is in Spark. Where I can find some reference? I searched and couldn't find. thanks
– Felipe
Mar 23 at 10:47
1
Take a look at this answer by Jacek Laskowski.
– user10958683
Mar 23 at 11:44
1
1
If the question is if the API is intended to be extended by the end users, then the answer is most likely negative. And to quote Felix Cheung response to acceleration aware scheduling - Third, the proposal touches on some fairly core and sensitive components, like the scheduler, and I think more discussions are necessary. We have a few comments there and in the JIRA.. Have you rather looked at prefferedLocations?
– user10958683
Mar 23 at 10:22
If the question is if the API is intended to be extended by the end users, then the answer is most likely negative. And to quote Felix Cheung response to acceleration aware scheduling - Third, the proposal touches on some fairly core and sensitive components, like the scheduler, and I think more discussions are necessary. We have a few comments there and in the JIRA.. Have you rather looked at prefferedLocations?
– user10958683
Mar 23 at 10:22
Hi @user10958683, thanks for your answer. Seems Flink also have this requirement (docs.google.com/document/d/…). About "prefferedLocations", I don't know what it is in Spark. Where I can find some reference? I searched and couldn't find. thanks
– Felipe
Mar 23 at 10:47
Hi @user10958683, thanks for your answer. Seems Flink also have this requirement (docs.google.com/document/d/…). About "prefferedLocations", I don't know what it is in Spark. Where I can find some reference? I searched and couldn't find. thanks
– Felipe
Mar 23 at 10:47
1
1
Take a look at this answer by Jacek Laskowski.
– user10958683
Mar 23 at 11:44
Take a look at this answer by Jacek Laskowski.
– user10958683
Mar 23 at 11:44
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55312586%2fis-it-possible-to-extend-the-scheduler-of-apache-spark%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55312586%2fis-it-possible-to-extend-the-scheduler-of-apache-spark%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
If the question is if the API is intended to be extended by the end users, then the answer is most likely negative. And to quote Felix Cheung response to acceleration aware scheduling - Third, the proposal touches on some fairly core and sensitive components, like the scheduler, and I think more discussions are necessary. We have a few comments there and in the JIRA.. Have you rather looked at prefferedLocations?
– user10958683
Mar 23 at 10:22
Hi @user10958683, thanks for your answer. Seems Flink also have this requirement (docs.google.com/document/d/…). About "prefferedLocations", I don't know what it is in Spark. Where I can find some reference? I searched and couldn't find. thanks
– Felipe
Mar 23 at 10:47
1
Take a look at this answer by Jacek Laskowski.
– user10958683
Mar 23 at 11:44