Styjun

How much advanced notice is expected for faculty interviews?

How do we know that black holes are spinning?

Olympic game scoring

ColorFunction based on array index in ListLinePlot

What's the benefit of prohibiting the use of techniques/language constructs that have not been taught?

How to publish superseding results without creating enemies

Why is the year in this ISO timestamp not 2019?

What exactly is a marshrutka (маршрутка)?

Are space camera sensors usually round, or square?

Does a succubus' charm end when it dies?

Python web-scraper to download table of transistor counts from Wikipedia

Consonance v. Dissonance

What is the mathematical notation for rounding a given number to the nearest integer?

How To Make Earth's Oceans as Brackish as Lyr's

Output a Super Mario Image

How do certain apps show new notifications when internet access is restricted to them?

Is low emotional intelligence associated with right-wing and prejudiced attitudes?

How does a simple logistic regression model achieve a 92% classification accuracy on MNIST?

POSIX compatible way to get user name associated with a user ID

Are there any rules about taking damage whilst holding your breath in combat?

Can I fix my boots by gluing the soles back on?

Is "you will become a subject matter expert" code for "you'll be working on your own 100% of the time"?

How does a linear operator act on a bra?

Which is the current decimal separator?

Is it possible to add Fair scheduler pools programmatically in Spark?

Apache Spark: How application level scheduling is done in spark standalone mode?How to enable Fair scheduler?Submitting Spark Job On Scheduler PoolIs FAIR available for Spark Standalone cluster mode?(Spark Schedular) What's the difference between Fair and FIFO inside a spark job pool?Spark fair scheduling not workingSpark Application FAIR SchedulingHow to correctly setup spark fair scheduler and pools?How Do I Enable Fair Scheduler in PySpark?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

I'm developing an application where several users use the same SparkContext to launch their queries to a Spark Cluster.

As the Spark documentation states (https://spark.apache.org/docs/2.2.0/job-scheduling.html#fair-scheduler-pools), with the Fair scheduler, you can assign a different pool for every user and they'll get a fair share of the cluster resources but every pool will be set up with the default pool configuration (scheduling mode FIFO, weight 1, and minShare 0).

Given that we don't know in advance which users can connect to the application, we can't set up a configuration file for the fair scheduler pools for all the users.

So, in order to give a pool to every user dynamically and set up every pool with a FAIR scheduling mode, I think there might be 2 options:

1. Change the default pool behaviour in order to change the scheduling mode to FAIR. Is it possible? How?


2. Generate a scheduler pool dynamically and programmatically in order to add a scheduler pool when a user connects to the application for first time and that pool should be created with a FAIR scheduling mode. Is it possible? How?

Thanks in advance

apache-spark

share|improve this question

edited Mar 29 at 9:04

miguel0afd

asked Mar 28 at 11:15

miguel0afdmiguel0afd

6299 bronze badges

add a comment
 | 

0

I'm developing an application where several users use the same SparkContext to launch their queries to a Spark Cluster.

As the Spark documentation states (https://spark.apache.org/docs/2.2.0/job-scheduling.html#fair-scheduler-pools), with the Fair scheduler, you can assign a different pool for every user and they'll get a fair share of the cluster resources but every pool will be set up with the default pool configuration (scheduling mode FIFO, weight 1, and minShare 0).

Given that we don't know in advance which users can connect to the application, we can't set up a configuration file for the fair scheduler pools for all the users.

So, in order to give a pool to every user dynamically and set up every pool with a FAIR scheduling mode, I think there might be 2 options:

1. Change the default pool behaviour in order to change the scheduling mode to FAIR. Is it possible? How?


2. Generate a scheduler pool dynamically and programmatically in order to add a scheduler pool when a user connects to the application for first time and that pool should be created with a FAIR scheduling mode. Is it possible? How?

Thanks in advance

apache-spark

share|improve this question

edited Mar 29 at 9:04

miguel0afd

asked Mar 28 at 11:15

miguel0afdmiguel0afd

6299 bronze badges

add a comment
 | 

I'm developing an application where several users use the same SparkContext to launch their queries to a Spark Cluster.

As the Spark documentation states (https://spark.apache.org/docs/2.2.0/job-scheduling.html#fair-scheduler-pools), with the Fair scheduler, you can assign a different pool for every user and they'll get a fair share of the cluster resources but every pool will be set up with the default pool configuration (scheduling mode FIFO, weight 1, and minShare 0).

Given that we don't know in advance which users can connect to the application, we can't set up a configuration file for the fair scheduler pools for all the users.

So, in order to give a pool to every user dynamically and set up every pool with a FAIR scheduling mode, I think there might be 2 options:

1. Change the default pool behaviour in order to change the scheduling mode to FAIR. Is it possible? How?


2. Generate a scheduler pool dynamically and programmatically in order to add a scheduler pool when a user connects to the application for first time and that pool should be created with a FAIR scheduling mode. Is it possible? How?

Thanks in advance

apache-spark

share|improve this question

edited Mar 29 at 9:04

miguel0afd

asked Mar 28 at 11:15

miguel0afdmiguel0afd

6299 bronze badges

share|improve this question

edited Mar 29 at 9:04

miguel0afd

asked Mar 28 at 11:15

miguel0afdmiguel0afd

6299 bronze badges

share|improve this question

edited Mar 29 at 9:04

miguel0afd

asked Mar 28 at 11:15

miguel0afdmiguel0afd

6299 bronze badges

asked Mar 28 at 11:15

miguel0afdmiguel0afd

6299 bronze badges

asked Mar 28 at 11:15

miguel0afdmiguel0afd

6299 bronze badges