AirflowException: SSH operator error: No authentication methods availableHow to set up emr_default in airflowApache Airflow — How to Specify Unix User when Using BashOperatorSFTPOperator in Airflow not working with a conn_idAirflow Exception: Dataflow failed with return code 2Job submit with SparkSubmitOperator which uses ssh for conn_typeEMR DAG terminates before all steps are completedAirflow SparkSubmitOperator - How to spark-submit in another serverHow to submit Spark jobs to EMR cluster from Airflow?Can't use plugin module with Cloud ComposerWhere to store credentials for AWS EMR Apache Spark application submitted from Airflow task
What's it called when the bad guy gets eaten?
What happens to unproductive professors?
Elf (adjective) vs. Elvish vs. Elven
Why archangel Michael didn't save Jesus when he was crucified?
Why does the Antonov AN-225 not have any winglets?
Is "I do not want you to go nowhere" a case of "DOUBLE-NEGATIVES" as claimed by Grammarly?
Why is the ladder of the LM always in the dark side of the LM?
Credit score and financing new car
How effective would wooden scale armor be in a medieval setting?
GDPR rights when subject dies; does family inherit subject rights?
What's the point of having a RAID 1 configuration over incremental backups to a secondary drive?
Is there any reason why MCU changed the Snap to Blip
Is a request to book a business flight ticket for a graduate student an unreasonable one?
Received a dinner invitation through my employer's email, is it ok to attend?
Graduate student with abysmal English writing skills, how to help
What is /bin/red
Could you brine steak?
How can a dictatorship government be beneficial to a dictator in a post-scarcity society?
LMOP: what beasts live in Neverwinter Wood?
What is this little owl-like bird?
[Future]Historical experience as a guide to warship design?
What are some further readings in Econometrics you recommend?
When I press the space bar it deletes the letters in front of it
Data Encryption by Application vs Data Encryption in Database
AirflowException: SSH operator error: No authentication methods available
How to set up emr_default in airflowApache Airflow — How to Specify Unix User when Using BashOperatorSFTPOperator in Airflow not working with a conn_idAirflow Exception: Dataflow failed with return code 2Job submit with SparkSubmitOperator which uses ssh for conn_typeEMR DAG terminates before all steps are completedAirflow SparkSubmitOperator - How to spark-submit in another serverHow to submit Spark jobs to EMR cluster from Airflow?Can't use plugin module with Cloud ComposerWhere to store credentials for AWS EMR Apache Spark application submitted from Airflow task
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I've been using Apache Airflow for a while now and due to a use-case I need to re-write the SparkSubmitOperator to SSHOperator. My airflow is installed on AWS EMR and the spark-submit needs to happen on the same EMR. But the operator fails with - AirflowException: SSH operator error: No authentication methods available
I'm using the default ssh_conn_id=ssh_default and tried passing as ssh_hook as well, but nothing seems to work
dag-file
hook = SSHHook(ssh_conn_id='ssh_default')
spark_submit = """
spark-submit ....
"""
process = SSHOperator(
task_id='execute_pySpark',
ssh_hook=hook,
command=spark_submit,
do_xcom_push=True
)
airflow connection
Conn Id: ssh_default
Conn Type: SSH
Host: localhost
Any insight is appreciated. Thank you in advance!
airflow
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
add a comment |
I've been using Apache Airflow for a while now and due to a use-case I need to re-write the SparkSubmitOperator to SSHOperator. My airflow is installed on AWS EMR and the spark-submit needs to happen on the same EMR. But the operator fails with - AirflowException: SSH operator error: No authentication methods available
I'm using the default ssh_conn_id=ssh_default and tried passing as ssh_hook as well, but nothing seems to work
dag-file
hook = SSHHook(ssh_conn_id='ssh_default')
spark_submit = """
spark-submit ....
"""
process = SSHOperator(
task_id='execute_pySpark',
ssh_hook=hook,
command=spark_submit,
do_xcom_push=True
)
airflow connection
Conn Id: ssh_default
Conn Type: SSH
Host: localhost
Any insight is appreciated. Thank you in advance!
airflow
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
add a comment |
I've been using Apache Airflow for a while now and due to a use-case I need to re-write the SparkSubmitOperator to SSHOperator. My airflow is installed on AWS EMR and the spark-submit needs to happen on the same EMR. But the operator fails with - AirflowException: SSH operator error: No authentication methods available
I'm using the default ssh_conn_id=ssh_default and tried passing as ssh_hook as well, but nothing seems to work
dag-file
hook = SSHHook(ssh_conn_id='ssh_default')
spark_submit = """
spark-submit ....
"""
process = SSHOperator(
task_id='execute_pySpark',
ssh_hook=hook,
command=spark_submit,
do_xcom_push=True
)
airflow connection
Conn Id: ssh_default
Conn Type: SSH
Host: localhost
Any insight is appreciated. Thank you in advance!
airflow
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
I've been using Apache Airflow for a while now and due to a use-case I need to re-write the SparkSubmitOperator to SSHOperator. My airflow is installed on AWS EMR and the spark-submit needs to happen on the same EMR. But the operator fails with - AirflowException: SSH operator error: No authentication methods available
I'm using the default ssh_conn_id=ssh_default and tried passing as ssh_hook as well, but nothing seems to work
dag-file
hook = SSHHook(ssh_conn_id='ssh_default')
spark_submit = """
spark-submit ....
"""
process = SSHOperator(
task_id='execute_pySpark',
ssh_hook=hook,
command=spark_submit,
do_xcom_push=True
)
airflow connection
Conn Id: ssh_default
Conn Type: SSH
Host: localhost
Any insight is appreciated. Thank you in advance!
airflow
airflow
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
asked Mar 26 at 0:50
Krunal PatelKrunal Patel
62 bronze badges
62 bronze badges
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Configure your SSH Connection as per the details specified at https://airflow.readthedocs.io/en/1.10.2/howto/manage-connections.html?highlight=SSHOperator#ssh
Add the username and password if there is one. The Host field should contain the IP address of your remote VM unless your Spark Master is on localhost.
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55348366%2fairflowexception-ssh-operator-error-no-authentication-methods-available%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Configure your SSH Connection as per the details specified at https://airflow.readthedocs.io/en/1.10.2/howto/manage-connections.html?highlight=SSHOperator#ssh
Add the username and password if there is one. The Host field should contain the IP address of your remote VM unless your Spark Master is on localhost.
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
add a comment |
Configure your SSH Connection as per the details specified at https://airflow.readthedocs.io/en/1.10.2/howto/manage-connections.html?highlight=SSHOperator#ssh
Add the username and password if there is one. The Host field should contain the IP address of your remote VM unless your Spark Master is on localhost.
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
add a comment |
Configure your SSH Connection as per the details specified at https://airflow.readthedocs.io/en/1.10.2/howto/manage-connections.html?highlight=SSHOperator#ssh
Add the username and password if there is one. The Host field should contain the IP address of your remote VM unless your Spark Master is on localhost.
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
Configure your SSH Connection as per the details specified at https://airflow.readthedocs.io/en/1.10.2/howto/manage-connections.html?highlight=SSHOperator#ssh
Add the username and password if there is one. The Host field should contain the IP address of your remote VM unless your Spark Master is on localhost.
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
answered Mar 26 at 10:12
kaxilkaxil
4,83011 silver badges31 bronze badges
4,83011 silver badges31 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55348366%2fairflowexception-ssh-operator-error-no-authentication-methods-available%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
