How does slot sharing help Flink?Dynamic number of jobs in Apache Flink - dealing with task slotsFlink State backend keys atomicy and distributiontaskmanager killed on Flink clusterWhat are the Apache Flink non-intensive vs intensive tasks?How does Flink handle serialization of managed state?Apache Flink: guideliness for setting parallelism?Apache Flink number of taskmanagers in local modeFlink job/data flow on multiple task managers failure and recoveryWhat are reasons to prefer increasing the number of task managers instead of task slots per task manager?Apache flink - PartitionNotFoundException
How much of data wrangling is a data scientist's job?
How can I tell someone that I want to be his or her friend?
In Romance of the Three Kingdoms why do people still use bamboo sticks when paper had already been invented?
CEO ridiculed me with gay jokes and grabbed me and wouldn't let go - now getting pushed out of company
Do I have a twin with permutated remainders?
Today is the Center
Infinite Abelian subgroup of infinite non Abelian group example
Is the Joker left-handed?
Emailing HOD to enhance faculty application
What is the intuition behind short exact sequences of groups; in particular, what is the intuition behind group extensions?
What mechanic is there to disable a threat instead of killing it?
Is it possible to create light that imparts a greater proportion of its energy as momentum rather than heat?
Where does SFDX store details about scratch orgs?
How to prevent "they're falling in love" trope
How do conventional missiles fly?
What killed these X2 caps?
In a spin, are both wings stalled?
What is a clear way to write a bar that has an extra beat?
Stopping power of mountain vs road bike
I'm flying to France today and my passport expires in less than 2 months
Can a virus destroy the BIOS of a modern computer?
Why doesn't using multiple commands with a || or && conditional work?
Arrow those variables!
How to take photos in burst mode, without vibration?
How does slot sharing help Flink?
Dynamic number of jobs in Apache Flink - dealing with task slotsFlink State backend keys atomicy and distributiontaskmanager killed on Flink clusterWhat are the Apache Flink non-intensive vs intensive tasks?How does Flink handle serialization of managed state?Apache Flink: guideliness for setting parallelism?Apache Flink number of taskmanagers in local modeFlink job/data flow on multiple task managers failure and recoveryWhat are reasons to prefer increasing the number of task managers instead of task slots per task manager?Apache flink - PartitionNotFoundException
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
Reading about Flink, what exactly are the benefits of slot sharing, for example why would I want to isolate slots in a Flink job?
My thinking is, assuming a 4GB JVM task manager, if I seperate this into two task slots, one called ts1 and another, ts2, I can put a very intensive windowing operation in ts1 while some map, filter etc can go into ts2?
apache-flink
asked Mar 21 at 22:01
dfdfdfdf
377
add a comment |
Reading about Flink, what exactly are the benefits of slot sharing, for example why would I want to isolate slots in a Flink job?
My thinking is, assuming a 4GB JVM task manager, if I seperate this into two task slots, one called ts1 and another, ts2, I can put a very intensive windowing operation in ts1 while some map, filter etc can go into ts2?
apache-flink
asked Mar 21 at 22:01
dfdfdfdf
377
add a comment |
Reading about Flink, what exactly are the benefits of slot sharing, for example why would I want to isolate slots in a Flink job?
My thinking is, assuming a 4GB JVM task manager, if I seperate this into two task slots, one called ts1 and another, ts2, I can put a very intensive windowing operation in ts1 while some map, filter etc can go into ts2?
apache-flink
asked Mar 21 at 22:01
dfdfdfdf
377
Reading about Flink, what exactly are the benefits of slot sharing, for example why would I want to isolate slots in a Flink job?
My thinking is, assuming a 4GB JVM task manager, if I seperate this into two task slots, one called ts1 and another, ts2, I can put a very intensive windowing operation in ts1 while some map, filter etc can go into ts2?
apache-flink
apache-flink
asked Mar 21 at 22:01
dfdfdfdf
377
asked Mar 21 at 22:01
dfdfdfdf
377
asked Mar 21 at 22:01
dfdfdfdf
377
asked Mar 21 at 22:01
dfdfdfdf
377
asked Mar 21 at 22:01
dfdfdfdf
377
377
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Slot sharing means that more than one sub-task is scheduled into the same slot -- or in other words, those operator instances end up sharing resources. This has these benefits:
- Better resource utilization. Otherwise you might easily end up with some slots doing very little work, while others are quite busy.
- Reduced network traffic.
- The number of slots then ends up being the highest degree of parallelism in the job. Having each slot run one parallel slice of the job makes it easier to reason about what's happening in the runtime.
You might find it advantageous to disable slot sharing if, as you point out, you want to devote more resources to an expensive operator. On the other hand, you could keep slot sharing enabled, and give each slot more cores and/or memory.
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
Thanks! Whats preferable, sharing slots and having isolated slots or increasing parallelism instead with 1 slot manager a task manager?
– dfdf
Mar 21 at 22:28
I don't understand this question.
– David Anderson
Mar 21 at 22:34
In terms of handling say an complicated operator, is it better to have the expensive operator in an isolated task group or have it run seperately in 100 taskamanagers with 1 task slot each/
– dfdf
Mar 21 at 22:38
Flink's scheduler treats all slots alike, so it doesn't make sense to have some slots that are more powerful than others.
– David Anderson
Mar 22 at 8:26
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55289922%2fhow-does-slot-sharing-help-flink%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Slot sharing means that more than one sub-task is scheduled into the same slot -- or in other words, those operator instances end up sharing resources. This has these benefits:
- Better resource utilization. Otherwise you might easily end up with some slots doing very little work, while others are quite busy.
- Reduced network traffic.
- The number of slots then ends up being the highest degree of parallelism in the job. Having each slot run one parallel slice of the job makes it easier to reason about what's happening in the runtime.
You might find it advantageous to disable slot sharing if, as you point out, you want to devote more resources to an expensive operator. On the other hand, you could keep slot sharing enabled, and give each slot more cores and/or memory.
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
Thanks! Whats preferable, sharing slots and having isolated slots or increasing parallelism instead with 1 slot manager a task manager?
– dfdf
Mar 21 at 22:28
I don't understand this question.
– David Anderson
Mar 21 at 22:34
In terms of handling say an complicated operator, is it better to have the expensive operator in an isolated task group or have it run seperately in 100 taskamanagers with 1 task slot each/
– dfdf
Mar 21 at 22:38
Flink's scheduler treats all slots alike, so it doesn't make sense to have some slots that are more powerful than others.
– David Anderson
Mar 22 at 8:26
add a comment |
Slot sharing means that more than one sub-task is scheduled into the same slot -- or in other words, those operator instances end up sharing resources. This has these benefits:
- Better resource utilization. Otherwise you might easily end up with some slots doing very little work, while others are quite busy.
- Reduced network traffic.
- The number of slots then ends up being the highest degree of parallelism in the job. Having each slot run one parallel slice of the job makes it easier to reason about what's happening in the runtime.
You might find it advantageous to disable slot sharing if, as you point out, you want to devote more resources to an expensive operator. On the other hand, you could keep slot sharing enabled, and give each slot more cores and/or memory.
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
Thanks! Whats preferable, sharing slots and having isolated slots or increasing parallelism instead with 1 slot manager a task manager?
– dfdf
Mar 21 at 22:28
I don't understand this question.
– David Anderson
Mar 21 at 22:34
In terms of handling say an complicated operator, is it better to have the expensive operator in an isolated task group or have it run seperately in 100 taskamanagers with 1 task slot each/
– dfdf
Mar 21 at 22:38
Flink's scheduler treats all slots alike, so it doesn't make sense to have some slots that are more powerful than others.
– David Anderson
Mar 22 at 8:26
add a comment |
Slot sharing means that more than one sub-task is scheduled into the same slot -- or in other words, those operator instances end up sharing resources. This has these benefits:
- Better resource utilization. Otherwise you might easily end up with some slots doing very little work, while others are quite busy.
- Reduced network traffic.
- The number of slots then ends up being the highest degree of parallelism in the job. Having each slot run one parallel slice of the job makes it easier to reason about what's happening in the runtime.
You might find it advantageous to disable slot sharing if, as you point out, you want to devote more resources to an expensive operator. On the other hand, you could keep slot sharing enabled, and give each slot more cores and/or memory.
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
Slot sharing means that more than one sub-task is scheduled into the same slot -- or in other words, those operator instances end up sharing resources. This has these benefits:
- Better resource utilization. Otherwise you might easily end up with some slots doing very little work, while others are quite busy.
- Reduced network traffic.
- The number of slots then ends up being the highest degree of parallelism in the job. Having each slot run one parallel slice of the job makes it easier to reason about what's happening in the runtime.
You might find it advantageous to disable slot sharing if, as you point out, you want to devote more resources to an expensive operator. On the other hand, you could keep slot sharing enabled, and give each slot more cores and/or memory.
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
edited Mar 21 at 22:32
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
answered Mar 21 at 22:20
David AndersonDavid Anderson
6,63921424
6,63921424
Thanks! Whats preferable, sharing slots and having isolated slots or increasing parallelism instead with 1 slot manager a task manager?
– dfdf
Mar 21 at 22:28
I don't understand this question.
– David Anderson
Mar 21 at 22:34
In terms of handling say an complicated operator, is it better to have the expensive operator in an isolated task group or have it run seperately in 100 taskamanagers with 1 task slot each/
– dfdf
Mar 21 at 22:38
Flink's scheduler treats all slots alike, so it doesn't make sense to have some slots that are more powerful than others.
– David Anderson
Mar 22 at 8:26
add a comment |
Thanks! Whats preferable, sharing slots and having isolated slots or increasing parallelism instead with 1 slot manager a task manager?
– dfdf
Mar 21 at 22:28
I don't understand this question.
– David Anderson
Mar 21 at 22:34
In terms of handling say an complicated operator, is it better to have the expensive operator in an isolated task group or have it run seperately in 100 taskamanagers with 1 task slot each/
– dfdf
Mar 21 at 22:38
Flink's scheduler treats all slots alike, so it doesn't make sense to have some slots that are more powerful than others.
– David Anderson
Mar 22 at 8:26
Thanks! Whats preferable, sharing slots and having isolated slots or increasing parallelism instead with 1 slot manager a task manager?
– dfdf
Mar 21 at 22:28
Thanks! Whats preferable, sharing slots and having isolated slots or increasing parallelism instead with 1 slot manager a task manager?
– dfdf
Mar 21 at 22:28
I don't understand this question.
– David Anderson
Mar 21 at 22:34
I don't understand this question.
– David Anderson
Mar 21 at 22:34
In terms of handling say an complicated operator, is it better to have the expensive operator in an isolated task group or have it run seperately in 100 taskamanagers with 1 task slot each/
– dfdf
Mar 21 at 22:38
In terms of handling say an complicated operator, is it better to have the expensive operator in an isolated task group or have it run seperately in 100 taskamanagers with 1 task slot each/
– dfdf
Mar 21 at 22:38
Flink's scheduler treats all slots alike, so it doesn't make sense to have some slots that are more powerful than others.
– David Anderson
Mar 22 at 8:26
Flink's scheduler treats all slots alike, so it doesn't make sense to have some slots that are more powerful than others.
– David Anderson
Mar 22 at 8:26
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55289922%2fhow-does-slot-sharing-help-flink%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
