Styjun

I am quite new to PySpark. Therefore this question may appear as quite elementary to others.
I am trying to export a data frame created via createOrReplaceTempView() to Hive. The steps are as follows

sqlcntx = SQLContext(sc)
df = sqlcntx.read.format("jdbc").options(url="sqlserver://.....details of MS Sql server",dbtable = "table_name").load()
df_cv_temp = df.createOrReplaceTempView("df")

When I use df_cv_temp.show(5) it is giving an error as follows

NoneType Object has no attribute 'show'

Interestingly when I try to see df.show(5) I am getting proper output.
Naturally when I see the above error I am not able to proceed further.

Now I have two questions.

1. How to fix the above issue?


2. Assuming the 1st issue is taken care of, what is the best way to export df_cv_temp to HIVE tables?

P.S. I am using PySaprk 2.0

Update: Incorporating Jim's Answer

Post answer received from Jim, I have updated the code. Please see below the revised code.

from pyspark.sql import HiveContext,SQLContext
sql_cntx = SQLContext(sc)
df = sqlcntx.read.format("jdbc").options(url="sqlserver://.....details of MS Sql server",dbtable = "table_name").load()
df_curr_volt.createOrReplaceTempView("df_cv_temp")
df_cv_filt = sql_cntx.sql("select * from df_cv_temp where DeviceTimeStamp between date_add(current_date(),-1) and current_date()") # Retrieving just a day's record
hc = HiveContext(sc)

Now the problem begins. Please refer to my question 2.

df_cv_tbl = hc.sql("create table if not exits df_cv_raw as select * from df_cv_filt")
df_cv_tbl.write.format("orc").saveAsTable("df_cv_raw")

The above two lines is producing the error as shown below.

pyspark.sql.utils.AnalysisException: u'Table or view not found: df_cv_filt; line 1 pos 14'

So what is the right way of approaching this?

Instead of

df_cv_temp = df.createOrReplaceTempView("df") 

you have to use,

df.createOrReplaceTempView("table1")

This is because, df.createOrReplaceTempView(<name_of_the_view>) creates (or replaces if that view name already exists) a lazily evaluated "view" that you can then use like a hive table in Spark SQL. The expression does not produce any output as such, hence it is a NoneType object.

Further, the temp view can be queried as below:

spark.sql("SELECT field1 AS f1, field2 as f2 from table1").show()

Incase, you are sure to have memory space, then you can persist it to be a hive table directly like below. This will create a managed Hive table physically; upon which you can query it even in your Hive CLI.

df.write.saveAsTable("table1")