Styjun

Are unclear "take-it or leave-it" contracts interpreted in my favor?

3D print appears to print very weak walls in long print

Is there any word for "disobedience to God"?

What does "un pareil soin" mean here?

Why does the U.S. tolerate foreign influence from Saudi Arabia and Israel on its domestic policies while not tolerating that from China or Russia?

Is "My neighbor Mr. Lee is 70 years old who plays basketball every day." grammatical?

Why does my String turn into Integers instead of letters after I add characters with +?

Is it worth upgrading to 28mm tyres from 25 for a 250 mile ride?

Is lack of functional requirements agile?

Should disabled buttons give feedback when clicked?

Why do people keep referring to Leia as Princess Leia, even after the destruction of Alderaan?

Find the last element of a list under conditions

Is there a word for a message that is intended to be intercepted by an adversary?

How would my creatures handle groups without a strong concept of numbers?

Matchmaker, Matchmaker, make me a match

Maximum charterer insertion

Are randomly-generated passwords starting with "a" less secure?

Why do players in the past play much longer tournaments than today's top players?

During copyediting, journal disagrees about spelling of paper's main topic

Is Trump personally blocking people on Twitter?

Why are they 'nude photos'?

What would be the ideal melee weapon made of "Phase Metal"?

Is Arc Length always irrational between two rational points?

If the railway suggests a 5-min connection window for changing trains in the Netherlands, does that mean it's definitely doable?

Scheduling Airflow operators on the same Kubernetes node with autoscaling

How to reduce nodes(vm) running in a Kubernetes cluster of GKE gracefully?GKE cluster autoscaler vs Autoscaling in Managed instance groupsHow does Kubernetes handle preemtible nodes going offline on GKE?kubelet restarting randomlyHow to best run Apache Airflow tasks on a Kubernetes cluster?Scheduling kube-dns on dedicated node poolPrevent Kubernetes rescheduling hundreds of pods when a node is failingAirflow Scheduler out of memory problemsKubernetes Node disconnected from masterkubernetes scheduling for expensive resources

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

1

I am using Airflow to compose and schedule long-running batch jobs. I hoped to take advantage of Kubernetes (GKE) cluster autoscaling by using either KubernetesExecutor or KubernetesPodOperator.

I played with GKE node pools and node selectors, and it seems to work as expected by scaling the number of worker nodes up and down to zero. However, some Airflow operators in my old pipeline share data via a file system, which should not work out of the box in Kubernetes as the pods may get scheduled on different nodes.

One solution I can think of is to fuse dependant operators in a single job like a one bash script but this way I lose the benefit of reusing existing Airflow operators. Another approach might be using something like NFS to share a staging directory between multiple workers. I suppose this might have performance and scalability issues.

QUESTION:
Is it possible to direct Kubernetes to schedule pods on the same node so that Airflow operators can share data via the host file system? I tried to use pod affinity for that, but in my case, it didn't help to get operators rescheduled on the same node.

kubernetes airflow google-kubernetes-engine

share|improve this question

edited Mar 26 at 4:57

kpax

asked Mar 26 at 3:18

kpaxkpax

14133 silver badges1010 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Did you try with antiAffinity/other affinities? banzaicloud.com/blog/k8s-affinities
  
  – Veerendra
  Mar 26 at 7:05
  
  

  
  
  
  

add a comment | 

0

1

I am using Airflow to compose and schedule long-running batch jobs. I hoped to take advantage of Kubernetes (GKE) cluster autoscaling by using either KubernetesExecutor or KubernetesPodOperator.

I played with GKE node pools and node selectors, and it seems to work as expected by scaling the number of worker nodes up and down to zero. However, some Airflow operators in my old pipeline share data via a file system, which should not work out of the box in Kubernetes as the pods may get scheduled on different nodes.

One solution I can think of is to fuse dependant operators in a single job like a one bash script but this way I lose the benefit of reusing existing Airflow operators. Another approach might be using something like NFS to share a staging directory between multiple workers. I suppose this might have performance and scalability issues.

QUESTION:
Is it possible to direct Kubernetes to schedule pods on the same node so that Airflow operators can share data via the host file system? I tried to use pod affinity for that, but in my case, it didn't help to get operators rescheduled on the same node.

kubernetes airflow google-kubernetes-engine

share|improve this question

edited Mar 26 at 4:57

kpax

asked Mar 26 at 3:18

kpaxkpax

14133 silver badges1010 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Did you try with antiAffinity/other affinities? banzaicloud.com/blog/k8s-affinities
  
  – Veerendra
  Mar 26 at 7:05
  
  

  
  
  
  

add a comment | 

I am using Airflow to compose and schedule long-running batch jobs. I hoped to take advantage of Kubernetes (GKE) cluster autoscaling by using either KubernetesExecutor or KubernetesPodOperator.

I played with GKE node pools and node selectors, and it seems to work as expected by scaling the number of worker nodes up and down to zero. However, some Airflow operators in my old pipeline share data via a file system, which should not work out of the box in Kubernetes as the pods may get scheduled on different nodes.

One solution I can think of is to fuse dependant operators in a single job like a one bash script but this way I lose the benefit of reusing existing Airflow operators. Another approach might be using something like NFS to share a staging directory between multiple workers. I suppose this might have performance and scalability issues.

QUESTION:
Is it possible to direct Kubernetes to schedule pods on the same node so that Airflow operators can share data via the host file system? I tried to use pod affinity for that, but in my case, it didn't help to get operators rescheduled on the same node.

kubernetes airflow google-kubernetes-engine

share|improve this question

edited Mar 26 at 4:57

kpax

asked Mar 26 at 3:18

kpaxkpax

14133 silver badges1010 bronze badges

kubernetes airflow google-kubernetes-engine

kubernetes airflow google-kubernetes-engine

share|improve this question

edited Mar 26 at 4:57

kpax

asked Mar 26 at 3:18

kpaxkpax

14133 silver badges1010 bronze badges

share|improve this question

edited Mar 26 at 4:57

kpax

asked Mar 26 at 3:18

kpaxkpax

14133 silver badges1010 bronze badges

asked Mar 26 at 3:18

kpaxkpax

14133 silver badges1010 bronze badges

asked Mar 26 at 3:18

kpaxkpax

14133 silver badges1010 bronze badges