Styjun

How do I find individuals that are participants in multiple (two) events?

Is this equation correct? And if so, is this famous?

Do Veracrypt encrypted volumes have any kind of brute force protection?

Should I worry about having my credit pulled multiple times while car shopping?

Idiom for 'person who gets violent when drunk"

Past vs. present tense when referring to a fictional character

Why does this Apple //e drops into system monitor when booting?

Realistic, logical way for men with medieval-era weaponry to compete with much larger and physically stronger foes

Can Dive Down protect a creature against Pacifism?

Can an open source licence be revoked if it violates employer's IP?

Parsing text written the millitext font

How to search for Android apps without ads?

I received a gift from my sister who just got back from

Approach sick days in feedback meeting

Why is my Taiyaki (Cake that looks like a fish) too hard and dry?

Manager wants to hire me; HR does not. How to proceed?

How can I find out about the game world without meta-influencing it?

How to represent jealousy in a cute way?

Was the Lonely Mountain, where Smaug lived, a volcano?

Why is it bad to use your whole foot in rock climbing

Nth term of Van Eck Sequence

ISP is not hashing the password I log in with online. Should I take any action?

Velocity of rotation of a sphere

Is fission/fusion to iron the most efficient way to convert mass to energy?

GCP Composer (Airflow) operator

setting up s3 for logs in airflowHow to validate airflow DAG with customer operator?How to force jinja templating on airflow variable?Migrate existing Airflow DB to Cloud ComposerCloud Composer (Airflow) jobs stuckCloud Composer GKE Node upgrade results in Airflow task randomly failingGCP, Composer, Airflow, OperatorsCreating Airflow DAGs on GCP ComposerRunning Hive queries with GCP ComposerAirflow run fails but airflow test works. Not sure why

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

3

I'm using the GCP Composer API (Airflow) and my DAG to scale up the number of workers keep returning me the error below:

Broken DAG: [/home/airflow/gcs/dags/cluster_scale_workers.py] 'module' object has no attribute 'DataProcClusterScaleOperator' 

Seems to be something related to the ScaleOperator, however when I look at the Airflow Read the Docs and cross check with my code, seems that nothing is wrong. What am I missing?

Is it related to GCP Airflow version?

Code:

import datetime
import os

from airflow import models
from airflow.contrib.operators import dataproc_operator
from airflow.utils import trigger_rule

yesterday = datetime.datetime.combine(
 datetime.datetime.today() - datetime.timedelta(1),
 datetime.datetime.min.time())

default_dag_args = 
 'start_date': yesterday,
 'email_on_failure': False,
 'email_on_retry': False,
 'retries': 1,
 'retry_delay': datetime.timedelta(minutes=5),
 'project_id': models.Variable.get('gcp_project'),
 'cluster_name': 'hive-cluster'


with models.DAG(
 'scale_workers',
 schedule_interval=datetime.timedelta(days=1),
 default_args=default_dag_args) as dag:

 scale_to_6_workers = dataproc_operator.DataprocClusterScaleOperator(
 task_id='scale_dataproc_cluster_6',
 cluster_name='hive-cluster',
 num_workers=6,
 num_preemptible_workers=3,
 dag=dag
 )

google-cloud-platform airflow google-cloud-dataproc google-cloud-composer

share|improve this question

edited Mar 25 at 0:57

Igor Dvorzhak

1,70311121

asked Dec 12 '18 at 13:52

RGreggRGregg

706

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Your code seems absolutely right. Which version of airflow your cloud composer is using? Try enabling BETA option at the top right corner of the cloud composer and check.
  
  – Ashish Kumar
  Dec 13 '18 at 8:52
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  (For Information) if you are not specifying the region of your cluster, by default it will take it as global.
  
  – Ashish Kumar
  Dec 13 '18 at 8:53
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks @AshishKumar, I noticed it was a version thing with my Airflow. I was using 1.9.0. Once I upgraded to 1.10.0, it worked fine.
  
  – RGregg
  Dec 13 '18 at 9:32
  

  
  
  
  

add a comment | 

3

I'm using the GCP Composer API (Airflow) and my DAG to scale up the number of workers keep returning me the error below:

Broken DAG: [/home/airflow/gcs/dags/cluster_scale_workers.py] 'module' object has no attribute 'DataProcClusterScaleOperator' 

Seems to be something related to the ScaleOperator, however when I look at the Airflow Read the Docs and cross check with my code, seems that nothing is wrong. What am I missing?

Is it related to GCP Airflow version?

Code:

import datetime
import os

from airflow import models
from airflow.contrib.operators import dataproc_operator
from airflow.utils import trigger_rule

yesterday = datetime.datetime.combine(
 datetime.datetime.today() - datetime.timedelta(1),
 datetime.datetime.min.time())

default_dag_args = 
 'start_date': yesterday,
 'email_on_failure': False,
 'email_on_retry': False,
 'retries': 1,
 'retry_delay': datetime.timedelta(minutes=5),
 'project_id': models.Variable.get('gcp_project'),
 'cluster_name': 'hive-cluster'


with models.DAG(
 'scale_workers',
 schedule_interval=datetime.timedelta(days=1),
 default_args=default_dag_args) as dag:

 scale_to_6_workers = dataproc_operator.DataprocClusterScaleOperator(
 task_id='scale_dataproc_cluster_6',
 cluster_name='hive-cluster',
 num_workers=6,
 num_preemptible_workers=3,
 dag=dag
 )

google-cloud-platform airflow google-cloud-dataproc google-cloud-composer

share|improve this question

edited Mar 25 at 0:57

Igor Dvorzhak

1,70311121

asked Dec 12 '18 at 13:52

RGreggRGregg

706

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Your code seems absolutely right. Which version of airflow your cloud composer is using? Try enabling BETA option at the top right corner of the cloud composer and check.
  
  – Ashish Kumar
  Dec 13 '18 at 8:52
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  (For Information) if you are not specifying the region of your cluster, by default it will take it as global.
  
  – Ashish Kumar
  Dec 13 '18 at 8:53
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks @AshishKumar, I noticed it was a version thing with my Airflow. I was using 1.9.0. Once I upgraded to 1.10.0, it worked fine.
  
  – RGregg
  Dec 13 '18 at 9:32
  

  
  
  
  

add a comment | 

I'm using the GCP Composer API (Airflow) and my DAG to scale up the number of workers keep returning me the error below:

Broken DAG: [/home/airflow/gcs/dags/cluster_scale_workers.py] 'module' object has no attribute 'DataProcClusterScaleOperator' 

Seems to be something related to the ScaleOperator, however when I look at the Airflow Read the Docs and cross check with my code, seems that nothing is wrong. What am I missing?

Is it related to GCP Airflow version?

Code:

import datetime
import os

from airflow import models
from airflow.contrib.operators import dataproc_operator
from airflow.utils import trigger_rule

yesterday = datetime.datetime.combine(
 datetime.datetime.today() - datetime.timedelta(1),
 datetime.datetime.min.time())

default_dag_args = 
 'start_date': yesterday,
 'email_on_failure': False,
 'email_on_retry': False,
 'retries': 1,
 'retry_delay': datetime.timedelta(minutes=5),
 'project_id': models.Variable.get('gcp_project'),
 'cluster_name': 'hive-cluster'


with models.DAG(
 'scale_workers',
 schedule_interval=datetime.timedelta(days=1),
 default_args=default_dag_args) as dag:

 scale_to_6_workers = dataproc_operator.DataprocClusterScaleOperator(
 task_id='scale_dataproc_cluster_6',
 cluster_name='hive-cluster',
 num_workers=6,
 num_preemptible_workers=3,
 dag=dag
 )

google-cloud-platform airflow google-cloud-dataproc google-cloud-composer

share|improve this question

edited Mar 25 at 0:57

Igor Dvorzhak

1,70311121

asked Dec 12 '18 at 13:52

RGreggRGregg

706

Broken DAG: [/home/airflow/gcs/dags/cluster_scale_workers.py] 'module' object has no attribute 'DataProcClusterScaleOperator' 

import datetime
import os

from airflow import models
from airflow.contrib.operators import dataproc_operator
from airflow.utils import trigger_rule

yesterday = datetime.datetime.combine(
 datetime.datetime.today() - datetime.timedelta(1),
 datetime.datetime.min.time())

default_dag_args = 
 'start_date': yesterday,
 'email_on_failure': False,
 'email_on_retry': False,
 'retries': 1,
 'retry_delay': datetime.timedelta(minutes=5),
 'project_id': models.Variable.get('gcp_project'),
 'cluster_name': 'hive-cluster'


with models.DAG(
 'scale_workers',
 schedule_interval=datetime.timedelta(days=1),
 default_args=default_dag_args) as dag:

 scale_to_6_workers = dataproc_operator.DataprocClusterScaleOperator(
 task_id='scale_dataproc_cluster_6',
 cluster_name='hive-cluster',
 num_workers=6,
 num_preemptible_workers=3,
 dag=dag
 )

share|improve this question

edited Mar 25 at 0:57

Igor Dvorzhak

1,70311121

asked Dec 12 '18 at 13:52

RGreggRGregg

706

share|improve this question

edited Mar 25 at 0:57

Igor Dvorzhak

1,70311121

asked Dec 12 '18 at 13:52

RGreggRGregg

706

edited Mar 25 at 0:57

Igor Dvorzhak

1,70311121

edited Mar 25 at 0:57

Igor Dvorzhak

1,70311121

asked Dec 12 '18 at 13:52

RGreggRGregg

706

asked Dec 12 '18 at 13:52

RGreggRGregg

706

1 Answer
1

active

oldest

votes

2

I managed to find the issue and sort it out. The comment provided by Ashish Kumar above is correct.

The problem was that the Airflow version I was using (1.9.0) did not support the DataProcClusterScaleOperator. I created another instance by activating BETA and choosing the version 1.10.0.

Which fixed my issue.

share|improve this answer

answered Dec 13 '18 at 9:34

RGreggRGregg

706

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53744563%2fgcp-composer-airflow-operator%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

I managed to find the issue and sort it out. The comment provided by Ashish Kumar above is correct.

The problem was that the Airflow version I was using (1.9.0) did not support the DataProcClusterScaleOperator. I created another instance by activating BETA and choosing the version 1.10.0.

Which fixed my issue.

share|improve this answer

answered Dec 13 '18 at 9:34

RGreggRGregg

706

add a comment | 

2

I managed to find the issue and sort it out. The comment provided by Ashish Kumar above is correct.

The problem was that the Airflow version I was using (1.9.0) did not support the DataProcClusterScaleOperator. I created another instance by activating BETA and choosing the version 1.10.0.

Which fixed my issue.

share|improve this answer

answered Dec 13 '18 at 9:34

RGreggRGregg

706

add a comment | 

I managed to find the issue and sort it out. The comment provided by Ashish Kumar above is correct.

The problem was that the Airflow version I was using (1.9.0) did not support the DataProcClusterScaleOperator. I created another instance by activating BETA and choosing the version 1.10.0.

Which fixed my issue.

share|improve this answer

answered Dec 13 '18 at 9:34

RGreggRGregg

706

share|improve this answer

answered Dec 13 '18 at 9:34

RGreggRGregg

706

answered Dec 13 '18 at 9:34

RGreggRGregg

706

answered Dec 13 '18 at 9:34

RGreggRGregg

706