Styjun

What verb goes with "coup"?

How can I get a file's size with C++17?

Why is 一日 used instead of 日 to ask how your day is?

What is the difference between a historical drama and a period drama?

Olive oil in Japanese cooking

Do I need to be legally qualified to install a Hive smart thermostat?

Finding integer database columns that may have their data type changed to reduce size

Why are symbols not written in words?

How long had Bertha Mason been in the attic at the point of the events in Jane Eyre

Which high-degree derivatives play an essential role?

Why is quantum gravity non-renormalizable?

Story about two rival crews terraforming a planet

Did Snape really give Umbridge a fake Veritaserum potion that Harry later pretended to drink?

Should I cross-validate metrics that were not optimised?

Should I cheat if the majority does it?

Are semivowels pronounced differently than vowels?

Why would a propellor have blades of different lengths?

Did Winston Churchill praise Rolls-Royce engines?

Term for a character that only exists to be talked to

Will greasing clutch parts make it softer

Contributing to a candidate as a Foreign National US Resident?

Is there any way for a Half-Orc Sorcerer to get proficiency with a heavy weapon?

Magento 2: I am not aware about magneto optimization. Can you please share the steps for this?

How come having a Deathly Hallow is not a big deal?

Unclear parse errors when translating complex PSQL query to SparkSQL

Kibana visualize - show most recent without aggregationEfficiently perform COUNT DISTINCT with spark, on csvs?Impala vs SparkSQL: built-in function translation: fnv_hashInserting into MySQL tables through SparkSQL, by querying from the same tableSparkSQL “CASE WHEN THEN” with two table columns in pysparkSPARK : Data frame Select expression failed to do append operation if both single quote and double quote present in the valueOptimize distinct values on a large number of columnsHow to concurrently insert SparkSQL query output to HIVE while using it for another SparkSQL queryNested query in SparkSQL with a LimitSpark Sql to read from Hive orc partitioned table giving array out of bound exception

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

I have been doing some analysis on the Lumen database and have recently swapped to spark as the csv's are >100GB and too hefty for a single machine.

Most of my queries are running fine, however, the following seems to be having some issues:

psql = "select b.*, "
"(select count(distinct c.notice_sender) from lumen_sender_duplicate_utility c where c.domain_name = b.domain_name and cast(c.num_of_dup_urls as int) = 0 ) num_of_distinct_senders "
"from (select a.domain_name, sum(a.num_of_url) total_num_urls, "
"sum(a.num_of_dup_urls) total_num_dup_urls, count(distinct a.notice_sender) total_num_senders " 
"from lumen_sender_duplicate a group by a.domain_name) b"

There have been a variety of errors as I have changed it around, but the most recent is the following: (full stack trace available at https://pastebin.com/raw/Xk4wVDmD)

Caused by: java.lang.RuntimeException: Couldn't find count(DISTINCT notice_sender)#419L in [domain_name#13,sum(cast(num_of_url#14 as double))#415,sum(cast(num_of_dup_urls#16 as double))#416,count(notice_sender#10)#417L]
 at scala.sys.package$.error(package.scala:27)

Initially I though this was because some features such as distinct or subqueries were not available, but I am using spark 2.4 and everything seems fine to that end. (I have also tested each of the components separately and there doesn't seem to be an issue). If anyone has an idea of where I am going wrong, any help would be much appreciated.

apache-spark pyspark apache-spark-sql pyspark-sql

share|improve this question

asked Mar 25 at 17:36

ben steerben steer

5777 bronze badges

add a comment | 

1

I have been doing some analysis on the Lumen database and have recently swapped to spark as the csv's are >100GB and too hefty for a single machine.

Most of my queries are running fine, however, the following seems to be having some issues:

psql = "select b.*, "
"(select count(distinct c.notice_sender) from lumen_sender_duplicate_utility c where c.domain_name = b.domain_name and cast(c.num_of_dup_urls as int) = 0 ) num_of_distinct_senders "
"from (select a.domain_name, sum(a.num_of_url) total_num_urls, "
"sum(a.num_of_dup_urls) total_num_dup_urls, count(distinct a.notice_sender) total_num_senders " 
"from lumen_sender_duplicate a group by a.domain_name) b"

There have been a variety of errors as I have changed it around, but the most recent is the following: (full stack trace available at https://pastebin.com/raw/Xk4wVDmD)

Caused by: java.lang.RuntimeException: Couldn't find count(DISTINCT notice_sender)#419L in [domain_name#13,sum(cast(num_of_url#14 as double))#415,sum(cast(num_of_dup_urls#16 as double))#416,count(notice_sender#10)#417L]
 at scala.sys.package$.error(package.scala:27)

Initially I though this was because some features such as distinct or subqueries were not available, but I am using spark 2.4 and everything seems fine to that end. (I have also tested each of the components separately and there doesn't seem to be an issue). If anyone has an idea of where I am going wrong, any help would be much appreciated.

apache-spark pyspark apache-spark-sql pyspark-sql

share|improve this question

asked Mar 25 at 17:36

ben steerben steer

5777 bronze badges

add a comment | 

I have been doing some analysis on the Lumen database and have recently swapped to spark as the csv's are >100GB and too hefty for a single machine.

Most of my queries are running fine, however, the following seems to be having some issues:

psql = "select b.*, "
"(select count(distinct c.notice_sender) from lumen_sender_duplicate_utility c where c.domain_name = b.domain_name and cast(c.num_of_dup_urls as int) = 0 ) num_of_distinct_senders "
"from (select a.domain_name, sum(a.num_of_url) total_num_urls, "
"sum(a.num_of_dup_urls) total_num_dup_urls, count(distinct a.notice_sender) total_num_senders " 
"from lumen_sender_duplicate a group by a.domain_name) b"

There have been a variety of errors as I have changed it around, but the most recent is the following: (full stack trace available at https://pastebin.com/raw/Xk4wVDmD)

Caused by: java.lang.RuntimeException: Couldn't find count(DISTINCT notice_sender)#419L in [domain_name#13,sum(cast(num_of_url#14 as double))#415,sum(cast(num_of_dup_urls#16 as double))#416,count(notice_sender#10)#417L]
 at scala.sys.package$.error(package.scala:27)

Initially I though this was because some features such as distinct or subqueries were not available, but I am using spark 2.4 and everything seems fine to that end. (I have also tested each of the components separately and there doesn't seem to be an issue). If anyone has an idea of where I am going wrong, any help would be much appreciated.

apache-spark pyspark apache-spark-sql pyspark-sql

share|improve this question

asked Mar 25 at 17:36

ben steerben steer

5777 bronze badges

psql = "select b.*, "
"(select count(distinct c.notice_sender) from lumen_sender_duplicate_utility c where c.domain_name = b.domain_name and cast(c.num_of_dup_urls as int) = 0 ) num_of_distinct_senders "
"from (select a.domain_name, sum(a.num_of_url) total_num_urls, "
"sum(a.num_of_dup_urls) total_num_dup_urls, count(distinct a.notice_sender) total_num_senders " 
"from lumen_sender_duplicate a group by a.domain_name) b"

Caused by: java.lang.RuntimeException: Couldn't find count(DISTINCT notice_sender)#419L in [domain_name#13,sum(cast(num_of_url#14 as double))#415,sum(cast(num_of_dup_urls#16 as double))#416,count(notice_sender#10)#417L]
 at scala.sys.package$.error(package.scala:27)

share|improve this question

asked Mar 25 at 17:36

ben steerben steer

5777 bronze badges

share|improve this question

asked Mar 25 at 17:36

ben steerben steer

5777 bronze badges

asked Mar 25 at 17:36

ben steerben steer

5777 bronze badges

asked Mar 25 at 17:36

ben steerben steer

5777 bronze badges