why airflow's webserver is exiting after filling up the DagbagAirflow “This DAG isnt available in the webserver DagBag object ”How does Airflow's BranchPythonOperator work?Airflow's Gunicorn is spamming error logsChanging an Airflow's DAG end_dateDeleting a SubDag from Airflow's databaseError using airflow's DataflowPythonOperator to schedule dataflow jobError while install airflow: By default one of Airflow's dependencies installs a GPLairflow 1.10.0 branchpythonoperator run failed : Celery command failedhow to resolve “Error: No module named 'airflow.www'” while starting airflow webseverApache - Airflow 1.10.1 don't start a job
Are there any instances of members of different Hogwarts houses coupling up and marrying each other?
SQL Server table with 4,000,000 rows is 40GB
Are there take-over requests from autopilots?
Is English tonal for some words, like "permit"?
Does the mana ability restriction of Pithing Needle refer to the cost or the effect of an activated ability?
Awesomism and its awesome gods
Why would "an mule" be used instead of "a mule"?
How to split a string by the third .(dot) delimiter
How can "life" insurance prevent the cheapening of death?
Why didn't Thor use the All powerful spear instead of Stormbreaker?
How does Vivi differ from other Black Mages?
Why are some Mac apps not available on AppStore?
How to circle together certain entries of a matrix?
Expected value until a success?
Tear out when plate making w/ a router
Are Democrats more likely to believe Astrology is a science?
What is this sticking out of my wall?
Show that there exists an integer that is a power of 5 with the first 4 digits being 2018.
What's the biggest difference between these two photos?
2.5 year old daughter refuses to take medicine
Guitar beginner - What does this card show?
What is this dollar sign ($) icon in my Menu Bar?
Is there a "right" way to interpret a novel? If so, how do we make sure our novel is interpreted correctly?
Could the government trigger by-elections to regain a majority?
why airflow's webserver is exiting after filling up the Dagbag
Airflow “This DAG isnt available in the webserver DagBag object ”How does Airflow's BranchPythonOperator work?Airflow's Gunicorn is spamming error logsChanging an Airflow's DAG end_dateDeleting a SubDag from Airflow's databaseError using airflow's DataflowPythonOperator to schedule dataflow jobError while install airflow: By default one of Airflow's dependencies installs a GPLairflow 1.10.0 branchpythonoperator run failed : Celery command failedhow to resolve “Error: No module named 'airflow.www'” while starting airflow webseverApache - Airflow 1.10.1 don't start a job
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I have been using airflow for a month without any such issues. But suddenly the webserver stopped working. It is exiting after filling up the DagBag. No errors are displayed either.
i even tried killing all the airflow and gunicorn processes. Still no luck.
This is what i see when executing "airflow webserver"
[uname@airflow airflow]$ airflow webserver
[2019-03-28 07:51:54,946] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=4128
[2019-03-28 07:51:55,356] __init__.py:51 INFO - Using executor LocalExecutor
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_http_token_header_name] not found in config
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_expected_http_token] not found in config
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/
[2019-03-28 07:51:55,705] models.py:273 INFO - Filling up the DagBag from /home/uname/airflow/dags
[uname@airflow airflow]$
airflow
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
add a comment |
I have been using airflow for a month without any such issues. But suddenly the webserver stopped working. It is exiting after filling up the DagBag. No errors are displayed either.
i even tried killing all the airflow and gunicorn processes. Still no luck.
This is what i see when executing "airflow webserver"
[uname@airflow airflow]$ airflow webserver
[2019-03-28 07:51:54,946] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=4128
[2019-03-28 07:51:55,356] __init__.py:51 INFO - Using executor LocalExecutor
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_http_token_header_name] not found in config
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_expected_http_token] not found in config
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/
[2019-03-28 07:51:55,705] models.py:273 INFO - Filling up the DagBag from /home/uname/airflow/dags
[uname@airflow airflow]$
airflow
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
1
Try running withlogging_levelset toDEBUGto see if you can get more logs.
– chris.mclennon
Mar 28 at 21:21
I figured the issue. I have put exit(1) somewhere in my code on a condition and so it exited the whole webserver while interpreting that code bit. Isn't that a bug? I just want my .py not to be loaded in the DagBag on some condition. How do i achieve that?
– SpaceyBot
Mar 29 at 5:13
1
You could run your condition in your DAG file and have the .py filereturnearly rather than doing exit(1). I haven't done this myself so not totally confident, but this should at least make sure the sys.exit(1) doesn't propagate up to the webserver. I thought this issue may have been resolved in a newer version of Airflow, so you can look into an update if you're not already on 1.10.x
– chris.mclennon
Mar 29 at 21:56
I'm already using 1.10.2. anyway, I'll try "return" way. thanks tho
– SpaceyBot
Mar 30 at 5:36
add a comment |
I have been using airflow for a month without any such issues. But suddenly the webserver stopped working. It is exiting after filling up the DagBag. No errors are displayed either.
i even tried killing all the airflow and gunicorn processes. Still no luck.
This is what i see when executing "airflow webserver"
[uname@airflow airflow]$ airflow webserver
[2019-03-28 07:51:54,946] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=4128
[2019-03-28 07:51:55,356] __init__.py:51 INFO - Using executor LocalExecutor
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_http_token_header_name] not found in config
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_expected_http_token] not found in config
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/
[2019-03-28 07:51:55,705] models.py:273 INFO - Filling up the DagBag from /home/uname/airflow/dags
[uname@airflow airflow]$
airflow
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
I have been using airflow for a month without any such issues. But suddenly the webserver stopped working. It is exiting after filling up the DagBag. No errors are displayed either.
i even tried killing all the airflow and gunicorn processes. Still no luck.
This is what i see when executing "airflow webserver"
[uname@airflow airflow]$ airflow webserver
[2019-03-28 07:51:54,946] settings.py:174 INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=4128
[2019-03-28 07:51:55,356] __init__.py:51 INFO - Using executor LocalExecutor
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_http_token_header_name] not found in config
[2019-03-28 07:51:55,459] configuration.py:255 WARNING - section/key [rest_api_plugin/rest_api_plugin_expected_http_token] not found in config
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/
[2019-03-28 07:51:55,705] models.py:273 INFO - Filling up the DagBag from /home/uname/airflow/dags
[uname@airflow airflow]$
airflow
airflow
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
asked Mar 28 at 7:54
SpaceyBotSpaceyBot
1148 bronze badges
1148 bronze badges
1
Try running withlogging_levelset toDEBUGto see if you can get more logs.
– chris.mclennon
Mar 28 at 21:21
I figured the issue. I have put exit(1) somewhere in my code on a condition and so it exited the whole webserver while interpreting that code bit. Isn't that a bug? I just want my .py not to be loaded in the DagBag on some condition. How do i achieve that?
– SpaceyBot
Mar 29 at 5:13
1
You could run your condition in your DAG file and have the .py filereturnearly rather than doing exit(1). I haven't done this myself so not totally confident, but this should at least make sure the sys.exit(1) doesn't propagate up to the webserver. I thought this issue may have been resolved in a newer version of Airflow, so you can look into an update if you're not already on 1.10.x
– chris.mclennon
Mar 29 at 21:56
I'm already using 1.10.2. anyway, I'll try "return" way. thanks tho
– SpaceyBot
Mar 30 at 5:36
add a comment |
1
Try running withlogging_levelset toDEBUGto see if you can get more logs.
– chris.mclennon
Mar 28 at 21:21
I figured the issue. I have put exit(1) somewhere in my code on a condition and so it exited the whole webserver while interpreting that code bit. Isn't that a bug? I just want my .py not to be loaded in the DagBag on some condition. How do i achieve that?
– SpaceyBot
Mar 29 at 5:13
1
You could run your condition in your DAG file and have the .py filereturnearly rather than doing exit(1). I haven't done this myself so not totally confident, but this should at least make sure the sys.exit(1) doesn't propagate up to the webserver. I thought this issue may have been resolved in a newer version of Airflow, so you can look into an update if you're not already on 1.10.x
– chris.mclennon
Mar 29 at 21:56
I'm already using 1.10.2. anyway, I'll try "return" way. thanks tho
– SpaceyBot
Mar 30 at 5:36
1
1
Try running with
logging_level set to DEBUG to see if you can get more logs.– chris.mclennon
Mar 28 at 21:21
Try running with
logging_level set to DEBUG to see if you can get more logs.– chris.mclennon
Mar 28 at 21:21
I figured the issue. I have put exit(1) somewhere in my code on a condition and so it exited the whole webserver while interpreting that code bit. Isn't that a bug? I just want my .py not to be loaded in the DagBag on some condition. How do i achieve that?
– SpaceyBot
Mar 29 at 5:13
I figured the issue. I have put exit(1) somewhere in my code on a condition and so it exited the whole webserver while interpreting that code bit. Isn't that a bug? I just want my .py not to be loaded in the DagBag on some condition. How do i achieve that?
– SpaceyBot
Mar 29 at 5:13
1
1
You could run your condition in your DAG file and have the .py file
return early rather than doing exit(1). I haven't done this myself so not totally confident, but this should at least make sure the sys.exit(1) doesn't propagate up to the webserver. I thought this issue may have been resolved in a newer version of Airflow, so you can look into an update if you're not already on 1.10.x– chris.mclennon
Mar 29 at 21:56
You could run your condition in your DAG file and have the .py file
return early rather than doing exit(1). I haven't done this myself so not totally confident, but this should at least make sure the sys.exit(1) doesn't propagate up to the webserver. I thought this issue may have been resolved in a newer version of Airflow, so you can look into an update if you're not already on 1.10.x– chris.mclennon
Mar 29 at 21:56
I'm already using 1.10.2. anyway, I'll try "return" way. thanks tho
– SpaceyBot
Mar 30 at 5:36
I'm already using 1.10.2. anyway, I'll try "return" way. thanks tho
– SpaceyBot
Mar 30 at 5:36
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55392562%2fwhy-airflows-webserver-is-exiting-after-filling-up-the-dagbag%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55392562%2fwhy-airflows-webserver-is-exiting-after-filling-up-the-dagbag%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Try running with
logging_levelset toDEBUGto see if you can get more logs.– chris.mclennon
Mar 28 at 21:21
I figured the issue. I have put exit(1) somewhere in my code on a condition and so it exited the whole webserver while interpreting that code bit. Isn't that a bug? I just want my .py not to be loaded in the DagBag on some condition. How do i achieve that?
– SpaceyBot
Mar 29 at 5:13
1
You could run your condition in your DAG file and have the .py file
returnearly rather than doing exit(1). I haven't done this myself so not totally confident, but this should at least make sure the sys.exit(1) doesn't propagate up to the webserver. I thought this issue may have been resolved in a newer version of Airflow, so you can look into an update if you're not already on 1.10.x– chris.mclennon
Mar 29 at 21:56
I'm already using 1.10.2. anyway, I'll try "return" way. thanks tho
– SpaceyBot
Mar 30 at 5:36