Styjun

Sub-subscripts in strings cause different spacings than subscripts

How many cones with angle theta can I pack into the unit sphere?

What does Linus Torvalds mean when he says that Git "never ever" tracks a file?

How to support a colleague who finds meetings extremely tiring?

Nested ellipses in tikzpicture: Chomsky hierarchy

Is it ok to offer lower paid work as a trial period before negotiating for a full-time job?

Why doesn't a hydraulic lever violate conservation of energy?

Windows 10: How to Lock (not sleep) laptop on lid close?

Could an empire control the whole planet with today's comunication methods?

How do you keep chess fun when your opponent constantly beats you?

Using dividends to reduce short term capital gains?

Why are PDP-7-style microprogrammed instructions out of vogue?

Did the new image of black hole confirm the general theory of relativity?

ELI5: Why do they say that Israel would have been the fourth country to land a spacecraft on the Moon and why do they call it low cost?

Huge performance difference of the command find with and without using %M option to show permissions

"... to apply for a visa" or "... and applied for a visa"?

Is every episode of "Where are my Pants?" identical?

Does Parliament hold absolute power in the UK?

Solving overdetermined system by QR decomposition

Do warforged have souls?

What do I do when my TA workload is more than expected?

Is an up-to-date browser secure on an out-of-date OS?

Accepted by European university, rejected by all American ones I applied to? Possible reasons?

Identify 80s or 90s comics with ripped creatures (not dwarves)

Spark-Scala application in Template Design Pattern

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

-3

I am developing a Spark-Scala application, in which I am planning to use Template Design Pattern.

Here is the proposed design.

ProjectTemplate.scala => This is a trait containing functions such as createSession, readData, processData, writeResult. Out of these 4, I am planning to give implementations for createSession, readData, writeResult in this trait, whereas the implementation for processData will be provided by the child class that implements this trait. I have tested this with giving println statements and the approach works.

So in all there are 3 components:

1) ProjectTemplate.scala =>

 trait ProjectTemplate 

 def createSession():sparkSession= 
 ... implementation is provided..
 

 def readData(): Dataset[Row] = 
 ... implementation is provided..
 

 def processData():Dataset[Row] //Implementation will be provided by child class

 def writeResult(result: Dataset[Row], filePath: String):Boolean=
 ... implementation is provided.. 
 

 def execute():Unit=
 createSession()
 readData()
 processData()
 writeResult()
 
 


2) Childclass.scala =>
 class Childclass extens ProjectTemplate
 override def processData(...):Dataset[Row]=
 <funciton implementatoin>
 ...
 
 


3) ChildObject.scala =>
 object ChildObject extens App 
 val obj = new Childclass
 obj.execute()
 

Command that I will use to submit the Spark application.

spark-submit --class package.ChildObject --master yarn --deploy-mode cluster

My question is: Will this application successfully create sparksession and return it. Can we do it this way? Thanks.

scala apache-spark design-patterns

share|improve this question

asked Mar 22 at 5:39

Don SamDon Sam

308

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Instead of downgrading the question, it would help if someone can comment on the correctness of incorrectness of the approach.
  
  – Don Sam
  Mar 22 at 14:49
  

  
  
  
  

add a comment | 

-3

I am developing a Spark-Scala application, in which I am planning to use Template Design Pattern.

Here is the proposed design.

ProjectTemplate.scala => This is a trait containing functions such as createSession, readData, processData, writeResult. Out of these 4, I am planning to give implementations for createSession, readData, writeResult in this trait, whereas the implementation for processData will be provided by the child class that implements this trait. I have tested this with giving println statements and the approach works.

So in all there are 3 components:

1) ProjectTemplate.scala =>

 trait ProjectTemplate 

 def createSession():sparkSession= 
 ... implementation is provided..
 

 def readData(): Dataset[Row] = 
 ... implementation is provided..
 

 def processData():Dataset[Row] //Implementation will be provided by child class

 def writeResult(result: Dataset[Row], filePath: String):Boolean=
 ... implementation is provided.. 
 

 def execute():Unit=
 createSession()
 readData()
 processData()
 writeResult()
 
 


2) Childclass.scala =>
 class Childclass extens ProjectTemplate
 override def processData(...):Dataset[Row]=
 <funciton implementatoin>
 ...
 
 


3) ChildObject.scala =>
 object ChildObject extens App 
 val obj = new Childclass
 obj.execute()
 

Command that I will use to submit the Spark application.

spark-submit --class package.ChildObject --master yarn --deploy-mode cluster

My question is: Will this application successfully create sparksession and return it. Can we do it this way? Thanks.

scala apache-spark design-patterns

share|improve this question

asked Mar 22 at 5:39

Don SamDon Sam

308

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Instead of downgrading the question, it would help if someone can comment on the correctness of incorrectness of the approach.
  
  – Don Sam
  Mar 22 at 14:49
  

  
  
  
  

add a comment | 

I am developing a Spark-Scala application, in which I am planning to use Template Design Pattern.

Here is the proposed design.

ProjectTemplate.scala => This is a trait containing functions such as createSession, readData, processData, writeResult. Out of these 4, I am planning to give implementations for createSession, readData, writeResult in this trait, whereas the implementation for processData will be provided by the child class that implements this trait. I have tested this with giving println statements and the approach works.

So in all there are 3 components:

1) ProjectTemplate.scala =>

 trait ProjectTemplate 

 def createSession():sparkSession= 
 ... implementation is provided..
 

 def readData(): Dataset[Row] = 
 ... implementation is provided..
 

 def processData():Dataset[Row] //Implementation will be provided by child class

 def writeResult(result: Dataset[Row], filePath: String):Boolean=
 ... implementation is provided.. 
 

 def execute():Unit=
 createSession()
 readData()
 processData()
 writeResult()
 
 


2) Childclass.scala =>
 class Childclass extens ProjectTemplate
 override def processData(...):Dataset[Row]=
 <funciton implementatoin>
 ...
 
 


3) ChildObject.scala =>
 object ChildObject extens App 
 val obj = new Childclass
 obj.execute()
 

Command that I will use to submit the Spark application.

spark-submit --class package.ChildObject --master yarn --deploy-mode cluster

My question is: Will this application successfully create sparksession and return it. Can we do it this way? Thanks.

scala apache-spark design-patterns

share|improve this question

asked Mar 22 at 5:39

Don SamDon Sam

308

1) ProjectTemplate.scala =>

 trait ProjectTemplate 

 def createSession():sparkSession= 
 ... implementation is provided..
 

 def readData(): Dataset[Row] = 
 ... implementation is provided..
 

 def processData():Dataset[Row] //Implementation will be provided by child class

 def writeResult(result: Dataset[Row], filePath: String):Boolean=
 ... implementation is provided.. 
 

 def execute():Unit=
 createSession()
 readData()
 processData()
 writeResult()
 
 


2) Childclass.scala =>
 class Childclass extens ProjectTemplate
 override def processData(...):Dataset[Row]=
 <funciton implementatoin>
 ...
 
 


3) ChildObject.scala =>
 object ChildObject extens App 
 val obj = new Childclass
 obj.execute()
 

share|improve this question

asked Mar 22 at 5:39

Don SamDon Sam

308

share|improve this question

asked Mar 22 at 5:39

Don SamDon Sam

308

asked Mar 22 at 5:39

Don SamDon Sam

308

asked Mar 22 at 5:39

Don SamDon Sam

308