Styjun

Why is the battery jumpered to a resistor in this schematic?

Unsolved Problems due to Lack of Computational Power

What does a comma signify in inorganic chemistry?

What if a restaurant suddenly cannot accept credit cards, and the customer has no cash?

Output the list of musical notes

Adding things to bunches of things vs multiplication

Understanding theorem 15.12 in Kosniovski's A first course in algebraic topology

The Lucky House

Can I use images from my published papers in my thesis without copyright infringment?

What was the intention with the Commodore 128?

Number of matrices with bounded products of rows and columns

May the tower use the runway while an emergency aircraft is inbound?

Ending a line of dialogue with "?!": Allowed or obnoxious?

Interaction between Leonin Warleader and Divine Visitation

When does The Truman Show take place?

Build a mob of suspiciously happy lenny faces ( ͡° ͜ʖ ͡°)

Do I need to start off my book by describing the character's "normal world"?

If it isn't [someone's name]!

Photoshop older default brushes

Are there any OR challenges that are similar to kaggle's competitions?

How do I answer an interview question about how to handle a hard deadline I won't be able to meet?

programming a recursive formula into Mathematica and find the nth position in the sequence

Parse a simple key=value config file in C

Have made several mistakes during the course of my PhD. Can't help but feel resentment. Can I get some advice about how to move forward?

Pyspark ALS recommendForAllUsers().first() hangs

Why do people write the #!/usr/bin/env python shebang on the first line of a Python script?How to remove the first Item from a list?importing pyspark in python shellHow to change dataframe column names in pyspark?Convert pyspark string to date formatFiltering a pyspark dataframe using isin by exclusionPySpark: when function with multiple outputsHow to solve Py4JJavaError:calling None.org.apache.spark.api.java.JavaSparkContextPyspark: find first occurrence of maximum valuePyspark hangs on simple command

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

I've created an ALS model and called its method .transform(test_data). I now want to view the predictions produced for the data.

userRecs.printSchema() produces:

 |-- ProductID: integer (nullable = false)
 |-- recommendations: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- CustomerID: integer (nullable = true)
 | | |-- rating: float (nullable = true)

Calling userRecs.first() results in the process hanging at 'Stage 4'

[Stage 4:> (0 + 1) / 1]

Am I handling/reading the data incorrectly? I'm also not sure why calling userRecs.first() requires more processing?

import pandas as pd
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS, ALSModel
from pyspark.ml.tuning import TrainValidationSplit, ParamGridBuilder
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from pyspark.sql.functions import explode

sc = SparkContext('local')
spark = SparkSession(sc)

# load the model
data = pd.read_csv('matric-out-small-SMALL.csv', sep=',')
df = spark.createDataFrame(data)
(training, test) = df.randomSplit([0.8, 0.2]) # seed , 50 
model = ALSModel.load("modelSaveOut")

# predict test ata
model.transform(test)

# Generate top 10 recommendations for each user
userRecs = model.recommendForAllUsers(3)

userRecs.printSchema()

userRecs.first()

Additionally, I would like to know if there another way to get the model to provide predictions for just a single data point?
I believe there is a better solution to get a prediction for a certain value)?

python apache-spark pyspark pyspark-sql apache-spark-dataset

share|improve this question

asked Mar 27 at 12:53

atomsatoms

2,21111 gold badge1212 silver badges2525 bronze badges

add a comment | 

0

I've created an ALS model and called its method .transform(test_data). I now want to view the predictions produced for the data.

userRecs.printSchema() produces:

 |-- ProductID: integer (nullable = false)
 |-- recommendations: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- CustomerID: integer (nullable = true)
 | | |-- rating: float (nullable = true)

Calling userRecs.first() results in the process hanging at 'Stage 4'

[Stage 4:> (0 + 1) / 1]

Am I handling/reading the data incorrectly? I'm also not sure why calling userRecs.first() requires more processing?

import pandas as pd
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS, ALSModel
from pyspark.ml.tuning import TrainValidationSplit, ParamGridBuilder
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from pyspark.sql.functions import explode

sc = SparkContext('local')
spark = SparkSession(sc)

# load the model
data = pd.read_csv('matric-out-small-SMALL.csv', sep=',')
df = spark.createDataFrame(data)
(training, test) = df.randomSplit([0.8, 0.2]) # seed , 50 
model = ALSModel.load("modelSaveOut")

# predict test ata
model.transform(test)

# Generate top 10 recommendations for each user
userRecs = model.recommendForAllUsers(3)

userRecs.printSchema()

userRecs.first()

Additionally, I would like to know if there another way to get the model to provide predictions for just a single data point?
I believe there is a better solution to get a prediction for a certain value)?

python apache-spark pyspark pyspark-sql apache-spark-dataset

share|improve this question

asked Mar 27 at 12:53

atomsatoms

2,21111 gold badge1212 silver badges2525 bronze badges

add a comment | 

I've created an ALS model and called its method .transform(test_data). I now want to view the predictions produced for the data.

userRecs.printSchema() produces:

 |-- ProductID: integer (nullable = false)
 |-- recommendations: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- CustomerID: integer (nullable = true)
 | | |-- rating: float (nullable = true)

Calling userRecs.first() results in the process hanging at 'Stage 4'

[Stage 4:> (0 + 1) / 1]

Am I handling/reading the data incorrectly? I'm also not sure why calling userRecs.first() requires more processing?

import pandas as pd
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS, ALSModel
from pyspark.ml.tuning import TrainValidationSplit, ParamGridBuilder
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from pyspark.sql.functions import explode

sc = SparkContext('local')
spark = SparkSession(sc)

# load the model
data = pd.read_csv('matric-out-small-SMALL.csv', sep=',')
df = spark.createDataFrame(data)
(training, test) = df.randomSplit([0.8, 0.2]) # seed , 50 
model = ALSModel.load("modelSaveOut")

# predict test ata
model.transform(test)

# Generate top 10 recommendations for each user
userRecs = model.recommendForAllUsers(3)

userRecs.printSchema()

userRecs.first()

Additionally, I would like to know if there another way to get the model to provide predictions for just a single data point?
I believe there is a better solution to get a prediction for a certain value)?

python apache-spark pyspark pyspark-sql apache-spark-dataset

share|improve this question

asked Mar 27 at 12:53

atomsatoms

2,21111 gold badge1212 silver badges2525 bronze badges

 |-- ProductID: integer (nullable = false)
 |-- recommendations: array (nullable = true)
 | |-- element: struct (containsNull = true)
 | | |-- CustomerID: integer (nullable = true)
 | | |-- rating: float (nullable = true)

import pandas as pd
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS, ALSModel
from pyspark.ml.tuning import TrainValidationSplit, ParamGridBuilder
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from pyspark.sql.functions import explode

sc = SparkContext('local')
spark = SparkSession(sc)

# load the model
data = pd.read_csv('matric-out-small-SMALL.csv', sep=',')
df = spark.createDataFrame(data)
(training, test) = df.randomSplit([0.8, 0.2]) # seed , 50 
model = ALSModel.load("modelSaveOut")

# predict test ata
model.transform(test)

# Generate top 10 recommendations for each user
userRecs = model.recommendForAllUsers(3)

userRecs.printSchema()

userRecs.first()

share|improve this question

asked Mar 27 at 12:53

atomsatoms

2,21111 gold badge1212 silver badges2525 bronze badges

share|improve this question

asked Mar 27 at 12:53

atomsatoms

2,21111 gold badge1212 silver badges2525 bronze badges

asked Mar 27 at 12:53

atomsatoms

2,21111 gold badge1212 silver badges2525 bronze badges

asked Mar 27 at 12:53

atomsatoms

2,21111 gold badge1212 silver badges2525 bronze badges