Styjun

Multi tool use

count network interfaces in bash

How to use FDE without needing to share the encryption password

Fermat's Last Theorem, mod n

What is the difference between turbojet and turbofan engines?

Is current (November 2019) polling about Democrats lead over Trump trustworthy?

Slimy whey in store-bought yoghurt

Get the last dates from multiple columns

My name causes an issue with any booking! (names end with MR and MRS)

Well-known American figure with Roman numerals

MS in Mathematics, having trouble finding work outside teaching algebra

Fivefold division of the whole tone - What does it mean?

Implement the Max-Pooling operation from Convolutional Neural Networks

'Nuke the sky' to make a rocket launch a tiny bit easier

Can we rotate symbols in LaTeX? How should we make this diagram?

Printing the bits of an integer using bitfields and union

How to get the address of a C++ lambda function within itself?

Doing 2 subsequent measurements of a Hadamard gate on the same qubit results every time in 0 on the 2nd one

What's the name of the role of characters who buff teammates?

How do I find the unknown program enabled during Start-Up?

Windows 10 ruined my GRUB menu

Will a nuclear country use nuclear weapons if attacked by conventional means by another nuclear country?

Selenium PageFactory vs Page Object Model

Why do airports in the UK have so few runways?

Fantasy movie with magical cliffs that crush a ship

How to execute python code and airflow macro in a BashOperator?

Macros in the Airflow Python operatorApache Airflow Best Practice: (Python)Operators or BashOperatorsHow do I stop an airflow DAG?Airflow scheduling issuesRun Airflow task at separate time from the rest of the DAG's tasksApache Airflow Xcom Pull from dynamic task nameAirflow how best to replace cron with airflowAirflow on_success_callback() does not runDAG executes OK from web interface, but “falls through” when run via command lineHow to get airflow time zone information from macros?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;

0

I am trying to execute the following task in Airflow:

time_zone = "America/New_York"
t1= BashOperator(
 task_id='example_task',
 bash_command=('script.sh --arg1 hello '
 f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
 dag=dag)

The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.

My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.

jinja2 airflow

share|improve this question

asked Mar 28 at 21:20

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something like echo execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) to see what that macro is actually resolving to.
  
  – chris.mclennon
  Mar 29 at 22:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  So you are saying that this execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) runs fine in BashOperator, where execution_date is an airflow macro and the rest is python code?
  
  – Newskooler
  Apr 1 at 23:46
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string to execution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')) , doing your stringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to change timedelta to macros.timedelta if you get a timedelta is undefined error.
  
  – chris.mclennon
  Apr 2 at 23:29
  
  

  
  
  
  

add a comment
 | 

0

I am trying to execute the following task in Airflow:

time_zone = "America/New_York"
t1= BashOperator(
 task_id='example_task',
 bash_command=('script.sh --arg1 hello '
 f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
 dag=dag)

The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.

My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.

jinja2 airflow

share|improve this question

asked Mar 28 at 21:20

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something like echo execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) to see what that macro is actually resolving to.
  
  – chris.mclennon
  Mar 29 at 22:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  So you are saying that this execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) runs fine in BashOperator, where execution_date is an airflow macro and the rest is python code?
  
  – Newskooler
  Apr 1 at 23:46
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string to execution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')) , doing your stringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to change timedelta to macros.timedelta if you get a timedelta is undefined error.
  
  – chris.mclennon
  Apr 2 at 23:29
  
  

  
  
  
  

add a comment
 | 

I am trying to execute the following task in Airflow:

time_zone = "America/New_York"
t1= BashOperator(
 task_id='example_task',
 bash_command=('script.sh --arg1 hello '
 f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
 dag=dag)

The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.

My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.

jinja2 airflow

share|improve this question

asked Mar 28 at 21:20

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

time_zone = "America/New_York"
t1= BashOperator(
 task_id='example_task',
 bash_command=('script.sh --arg1 hello '
 f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
 dag=dag)

share|improve this question

asked Mar 28 at 21:20

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

share|improve this question

asked Mar 28 at 21:20

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

asked Mar 28 at 21:20

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

asked Mar 28 at 21:20

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

1 Answer
1

active

oldest

votes

1

The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.

There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.

It's not the cleanest solution, but it works.

share|improve this answer

edited Apr 1 at 22:05

answered Apr 1 at 20:47

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

add a comment
 | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55407020%2fhow-to-execute-python-code-and-airflow-macro-in-a-bashoperator%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.

There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.

It's not the cleanest solution, but it works.

share|improve this answer

edited Apr 1 at 22:05

answered Apr 1 at 20:47

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

add a comment
 | 

1

The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.

There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.

It's not the cleanest solution, but it works.

share|improve this answer

edited Apr 1 at 22:05

answered Apr 1 at 20:47

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

add a comment
 | 

The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.

There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.

It's not the cleanest solution, but it works.

share|improve this answer

edited Apr 1 at 22:05

answered Apr 1 at 20:47

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

share|improve this answer

edited Apr 1 at 22:05

answered Apr 1 at 20:47

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

answered Apr 1 at 20:47

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges

answered Apr 1 at 20:47

NewskoolerNewskooler

92122 gold badges1414 silver badges3030 bronze badges