How to execute python code and airflow macro in a BashOperator?Macros in the Airflow Python operatorApache Airflow Best Practice: (Python)Operators or BashOperatorsHow do I stop an airflow DAG?Airflow scheduling issuesRun Airflow task at separate time from the rest of the DAG's tasksApache Airflow Xcom Pull from dynamic task nameAirflow how best to replace cron with airflowAirflow on_success_callback() does not runDAG executes OK from web interface, but “falls through” when run via command lineHow to get airflow time zone information from macros?
count network interfaces in bash
How to use FDE without needing to share the encryption password
Fermat's Last Theorem, mod n
What is the difference between turbojet and turbofan engines?
Is current (November 2019) polling about Democrats lead over Trump trustworthy?
Slimy whey in store-bought yoghurt
Get the last dates from multiple columns
My name causes an issue with any booking! (names end with MR and MRS)
Well-known American figure with Roman numerals
MS in Mathematics, having trouble finding work outside teaching algebra
Fivefold division of the whole tone - What does it mean?
Implement the Max-Pooling operation from Convolutional Neural Networks
'Nuke the sky' to make a rocket launch a tiny bit easier
Can we rotate symbols in LaTeX? How should we make this diagram?
Printing the bits of an integer using bitfields and union
How to get the address of a C++ lambda function within itself?
Doing 2 subsequent measurements of a Hadamard gate on the same qubit results every time in 0 on the 2nd one
What's the name of the role of characters who buff teammates?
How do I find the unknown program enabled during Start-Up?
Windows 10 ruined my GRUB menu
Will a nuclear country use nuclear weapons if attacked by conventional means by another nuclear country?
Selenium PageFactory vs Page Object Model
Why do airports in the UK have so few runways?
Fantasy movie with magical cliffs that crush a ship
How to execute python code and airflow macro in a BashOperator?
Macros in the Airflow Python operatorApache Airflow Best Practice: (Python)Operators or BashOperatorsHow do I stop an airflow DAG?Airflow scheduling issuesRun Airflow task at separate time from the rest of the DAG's tasksApache Airflow Xcom Pull from dynamic task nameAirflow how best to replace cron with airflowAirflow on_success_callback() does not runDAG executes OK from web interface, but “falls through” when run via command lineHow to get airflow time zone information from macros?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;
I am trying to execute the following task in Airflow:
time_zone = "America/New_York"
t1= BashOperator(
task_id='example_task',
bash_command=('script.sh --arg1 hello '
f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
dag=dag)
The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.
My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.
jinja2 airflow
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
add a comment
|
I am trying to execute the following task in Airflow:
time_zone = "America/New_York"
t1= BashOperator(
task_id='example_task',
bash_command=('script.sh --arg1 hello '
f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
dag=dag)
The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.
My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.
jinja2 airflow
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something likeecho execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))to see what that macro is actually resolving to.
– chris.mclennon
Mar 29 at 22:01
So you are saying that thisexecution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))runs fine inBashOperator, whereexecution_dateis an airflow macro and the rest is python code?
– Newskooler
Apr 1 at 23:46
I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string toexecution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')), doing yourstringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to changetimedeltatomacros.timedeltaif you get atimedelta is undefinederror.
– chris.mclennon
Apr 2 at 23:29
add a comment
|
I am trying to execute the following task in Airflow:
time_zone = "America/New_York"
t1= BashOperator(
task_id='example_task',
bash_command=('script.sh --arg1 hello '
f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
dag=dag)
The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.
My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.
jinja2 airflow
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
I am trying to execute the following task in Airflow:
time_zone = "America/New_York"
t1= BashOperator(
task_id='example_task',
bash_command=('script.sh --arg1 hello '
f'--arg2 execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) '),
dag=dag)
The problem I have is with the bash_command. I am trying to execute python code and use a macro inside the bash_command, but the script above yields airflow.exceptions.AirflowException: Bash command failed.
My question is: Is what I am trying to do possible and if so, how? I guess I am just scripting jinja in the wrong way... but I am not sure.
jinja2 airflow
jinja2 airflow
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
asked Mar 28 at 21:20
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
9212 gold badges14 silver badges30 bronze badges
I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something likeecho execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))to see what that macro is actually resolving to.
– chris.mclennon
Mar 29 at 22:01
So you are saying that thisexecution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))runs fine inBashOperator, whereexecution_dateis an airflow macro and the rest is python code?
– Newskooler
Apr 1 at 23:46
I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string toexecution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')), doing yourstringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to changetimedeltatomacros.timedeltaif you get atimedelta is undefinederror.
– chris.mclennon
Apr 2 at 23:29
add a comment
|
I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something likeecho execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))to see what that macro is actually resolving to.
– chris.mclennon
Mar 29 at 22:01
So you are saying that thisexecution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))runs fine inBashOperator, whereexecution_dateis an airflow macro and the rest is python code?
– Newskooler
Apr 1 at 23:46
I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string toexecution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')), doing yourstringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to changetimedeltatomacros.timedeltaif you get atimedelta is undefinederror.
– chris.mclennon
Apr 2 at 23:29
I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something like
echo execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) to see what that macro is actually resolving to.– chris.mclennon
Mar 29 at 22:01
I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something like
echo execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) to see what that macro is actually resolving to.– chris.mclennon
Mar 29 at 22:01
So you are saying that this
execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) runs fine in BashOperator, where execution_date is an airflow macro and the rest is python code?– Newskooler
Apr 1 at 23:46
So you are saying that this
execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone)) runs fine in BashOperator, where execution_date is an airflow macro and the rest is python code?– Newskooler
Apr 1 at 23:46
I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string to
execution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')) , doing your stringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to change timedelta to macros.timedelta if you get a timedelta is undefined error.– chris.mclennon
Apr 2 at 23:29
I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string to
execution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')) , doing your stringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to change timedelta to macros.timedelta if you get a timedelta is undefined error.– chris.mclennon
Apr 2 at 23:29
add a comment
|
1 Answer
1
active
oldest
votes
The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.
There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.
It's not the cleanest solution, but it works.
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
add a comment
|
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55407020%2fhow-to-execute-python-code-and-airflow-macro-in-a-bashoperator%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.
There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.
It's not the cleanest solution, but it works.
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
add a comment
|
The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.
There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.
It's not the cleanest solution, but it works.
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
add a comment
|
The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.
There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.
It's not the cleanest solution, but it works.
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
The reason why the above does not work is because I was using both jinja2 and python f-strings at the same time, thus resulting in confusion.
There is no way (which I have found) to combine the two directly from the bash_command=. A wrapper python function to execute the bash command and a PythonOperator to execute the wrapper function is a solution, as it provides great flexibility over the usage of the airflow macros (the reason why I use jinja2 in the bash_command= and python code.
It's not the cleanest solution, but it works.
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
edited Apr 1 at 22:05
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
answered Apr 1 at 20:47
NewskoolerNewskooler
9212 gold badges14 silver badges30 bronze badges
9212 gold badges14 silver badges30 bronze badges
add a comment
|
add a comment
|
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55407020%2fhow-to-execute-python-code-and-airflow-macro-in-a-bashoperator%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I've been able to run airflow macros in a bash script as you described, so I'm guessing the bash script itself may have failed somehow. You could further debug by running the bash image with something like
echo execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))to see what that macro is actually resolving to.– chris.mclennon
Mar 29 at 22:01
So you are saying that this
execution_date + timedelta(days=1).astimezone(pytz.timezone(time_zone))runs fine inBashOperator, whereexecution_dateis an airflow macro and the rest is python code?– Newskooler
Apr 1 at 23:46
I didn't try evaluating that exact string. I tried playing around with the jinja formatting and I'd recommend changing your string to
execution_date + timedelta(days=1).astimezone(pytz.timezone('time_zone')), doing yourstringvar.format(time_zone=time_zone), and then jinja should be able to resolve the execution_date from there. You might also want to changetimedeltatomacros.timedeltaif you get atimedelta is undefinederror.– chris.mclennon
Apr 2 at 23:29