Styjun

Does Ubuntu reduce battery life?

Word for giving preference to the oldest child

How do you deal with characters with multiple races?

How to prevent a single-element caster from being useless against immune foes?

"DDoouubbllee ssppeeaakk!!"

Rampant sharing of authorship among colleagues in the name of "collaboration". Is not taking part in it a death knell for a future in academia?

What language is Raven using for her attack in the new 52?

Why did some Apollo missions carry a grenade launcher?

How can flights operated by the same company have such different prices when marketed by another?

How does Asimov's second law deal with contradictory orders from different people?

Just how much information should you share with a former client?

Scam? Checks via Email

Coworker mumbles to herself when working, how to ask her to stop?

How can Paypal know my card is being used in another account?

Why don't short runways use ramps for takeoff?

Raindrops in Python

PCB design using code instead of clicking a mouse?

Can you continue the movement of a Bonus Action Dash granted by Expeditious Retreat if your Concentration is broken mid-move?

What are the closest international airports in different countries?

What does "in official capacity" mean?

On the sensitivity conjecture?

How can I convert a linear narrative into a branching narrative?

Applications of pure mathematics in operations research

Would people understand me speaking German all over Europe?

Running multiple tasks of one DAG in separate machines in Airflow

Airflow dynamic DAG and Task IdsAirflow: Tasks queued but not runningAirflow server not running jobsAirflow settings to run Celery workers in different Docker container?Airflow Scheduler and Webserver hangs while queuing the task to run on RabbitMQAirflow Task execution on multiple celery workerAirflow 1.9.0 is queuing but not launching tasksAirflow 1.9 - Tasks stuck in queueAirflow not picking up cleared tasks in complete dagIssues in running Airflow as cluster with Celery executors

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

I need to create a dag which looks like this-

print_date task needs to run from a server A and templated task needs to run from server B. From the documentation it is clear that celery with Redis or RabbitMq will be required. I am using celery along with Redis(puckel/docker-airflow). I already have airflow running in server B with celery executer.

Do I need to have the same setup in server A as well ??
Also, how will I connect these two tasks in a single dag which are actually present in the different server?
A sample framework for this kind of use case will be much appreciated.

redis celery airflow

share|improve this question

asked Mar 26 at 21:22

anarchyanarchy

34711 silver badge1414 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Do we need to have Redis on both machines
  
  – anarchy
  Apr 11 at 9:48
  
  

  
  
  
  

add a comment | 

0

I need to create a dag which looks like this-

print_date task needs to run from a server A and templated task needs to run from server B. From the documentation it is clear that celery with Redis or RabbitMq will be required. I am using celery along with Redis(puckel/docker-airflow). I already have airflow running in server B with celery executer.

Do I need to have the same setup in server A as well ??
Also, how will I connect these two tasks in a single dag which are actually present in the different server?
A sample framework for this kind of use case will be much appreciated.

redis celery airflow

share|improve this question

asked Mar 26 at 21:22

anarchyanarchy

34711 silver badge1414 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Do we need to have Redis on both machines
  
  – anarchy
  Apr 11 at 9:48
  
  

  
  
  
  

add a comment | 

I need to create a dag which looks like this-

print_date task needs to run from a server A and templated task needs to run from server B. From the documentation it is clear that celery with Redis or RabbitMq will be required. I am using celery along with Redis(puckel/docker-airflow). I already have airflow running in server B with celery executer.

Do I need to have the same setup in server A as well ??
Also, how will I connect these two tasks in a single dag which are actually present in the different server?
A sample framework for this kind of use case will be much appreciated.

redis celery airflow

share|improve this question

asked Mar 26 at 21:22

anarchyanarchy

34711 silver badge1414 bronze badges

share|improve this question

asked Mar 26 at 21:22

anarchyanarchy

34711 silver badge1414 bronze badges

share|improve this question

asked Mar 26 at 21:22

anarchyanarchy

34711 silver badge1414 bronze badges

asked Mar 26 at 21:22

anarchyanarchy

34711 silver badge1414 bronze badges

asked Mar 26 at 21:22

anarchyanarchy

34711 silver badge1414 bronze badges

1 Answer
1

active

oldest

votes

1

Use Airflow Queues. And when you define your task add a queue parameter and assign it to a particular queue.

For example, queue1 would just run all the task on Machine 1 & queue2 would run all tasks on Machine 2.

So you can assign your task A to queue 1, hence it would run on Machine 1
and assign

task B to queue 2, hence it would run on Machine 2

Check documentation at https://airflow.apache.org/concepts.html#queues

share|improve this answer

edited Mar 27 at 13:35

answered Mar 27 at 11:58

kaxilkaxil

5,0581111 silver badges3333 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  so i have trigger the dag from both the machines ??
  
  – anarchy
  Mar 27 at 12:42
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  I have updated my answer with an example
  
  – kaxil
  Mar 27 at 13:35
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This does not solve the problem. It's just a "bypass". Besides, to work with more than one queue, the queues must be explicitly specified in the task
  
  – JonyD
  May 17 at 16:18
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55366387%2frunning-multiple-tasks-of-one-dag-in-separate-machines-in-airflow%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

Use Airflow Queues. And when you define your task add a queue parameter and assign it to a particular queue.

For example, queue1 would just run all the task on Machine 1 & queue2 would run all tasks on Machine 2.

So you can assign your task A to queue 1, hence it would run on Machine 1
and assign

task B to queue 2, hence it would run on Machine 2

Check documentation at https://airflow.apache.org/concepts.html#queues

share|improve this answer

edited Mar 27 at 13:35

answered Mar 27 at 11:58

kaxilkaxil

5,0581111 silver badges3333 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  so i have trigger the dag from both the machines ??
  
  – anarchy
  Mar 27 at 12:42
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  I have updated my answer with an example
  
  – kaxil
  Mar 27 at 13:35
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This does not solve the problem. It's just a "bypass". Besides, to work with more than one queue, the queues must be explicitly specified in the task
  
  – JonyD
  May 17 at 16:18
  

  
  
  
  

add a comment | 

1

Use Airflow Queues. And when you define your task add a queue parameter and assign it to a particular queue.

For example, queue1 would just run all the task on Machine 1 & queue2 would run all tasks on Machine 2.

So you can assign your task A to queue 1, hence it would run on Machine 1
and assign

task B to queue 2, hence it would run on Machine 2

Check documentation at https://airflow.apache.org/concepts.html#queues

share|improve this answer

edited Mar 27 at 13:35

answered Mar 27 at 11:58

kaxilkaxil

5,0581111 silver badges3333 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  so i have trigger the dag from both the machines ??
  
  – anarchy
  Mar 27 at 12:42
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  I have updated my answer with an example
  
  – kaxil
  Mar 27 at 13:35
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This does not solve the problem. It's just a "bypass". Besides, to work with more than one queue, the queues must be explicitly specified in the task
  
  – JonyD
  May 17 at 16:18
  

  
  
  
  

add a comment | 

Use Airflow Queues. And when you define your task add a queue parameter and assign it to a particular queue.

For example, queue1 would just run all the task on Machine 1 & queue2 would run all tasks on Machine 2.

So you can assign your task A to queue 1, hence it would run on Machine 1
and assign

task B to queue 2, hence it would run on Machine 2

Check documentation at https://airflow.apache.org/concepts.html#queues

share|improve this answer

edited Mar 27 at 13:35

answered Mar 27 at 11:58

kaxilkaxil

5,0581111 silver badges3333 bronze badges

share|improve this answer

edited Mar 27 at 13:35

answered Mar 27 at 11:58

kaxilkaxil

5,0581111 silver badges3333 bronze badges

answered Mar 27 at 11:58

kaxilkaxil

5,0581111 silver badges3333 bronze badges

answered Mar 27 at 11:58

kaxilkaxil

5,0581111 silver badges3333 bronze badges