Styjun

Is it normal practice to screen share with a client?

Is my employer paying me fairly? Going from 1099 to W2

What do I do when a student working in my lab "ghosts" me?

Explanation for a joke about a three-legged dog that walks into a bar

Character is called by their first initial. How do I write it?

Is there anything wrong with Thrawn?

What are the exact meanings of roll, pitch and yaw?

How to copy a file transactionally?

Which Roman general was killed by his own soldiers for not letting them to loot a newly conquered city?

What exactly makes a General Products hull nearly indestructible?

Strange Cron Job takes up 100% of CPU Ubuntu 18 LTS Server

Is it legal for private citizens to "impound" e-scooters?

Magento2: How can I logout customer from controller?

On the strategic interest of giving long lasting stock orders

Why is a dedicated QA team member necessary?

What should I say when a company asks you why someone (a friend) who was fired left?

What to do when you reach a conclusion and find out later on that someone else already did?

Why is chess failing to attract big name sponsors?

A fictional island on Earth with "longer" springs and autumns

What do teaching faculty do during semester breaks?

What does "see" in "the Holy See" mean?

Automatic Habit of Meditation

Why isn't there a serious attempt at creating a third mass-appeal party in the US?

kids pooling money for Lego League and taxes

PySpark: lasso returns all coefficients non-zero

Pvalues of coefficients in Lasso in scikit-learnDetermining regression equation from coefficients obtained by Lasso Logistic Regression in python?Why does the lasso here didn't provide me with zero coefficient?Can I extract significane values for Logistic Regression coefficients in pysparkCoefficients and significance of lasso/ridgeR: Plotting lasso beta coefficientsLasso regression, generates a matrix of coefficientsPyspark Logistic Regression has zero coefficients after fittingcoefficients of logistic regression have no attribute indices in pysparkrandomSplit returns dataframe with all zero values in pyspark

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

0

After splitting my data in to training and testing, my training data has about 33 million records. I have 77 features and a binary response. I am fitting a logistic regression with lasso. However, lasso returns non-zero coefficient values for all the columns. I explored the correlation of the features among themselves and some of them have correlation of up to 0.8 and 0.7. Provided that I have many features (77) and the features are correlated, I would expect lasso to filter out at least some of the features. Why is this not happening? I could not share the data because of its size and privacy. But the code I am using is below. I am looking for what could be happening and what to explore.

logreg = LogisticRegression(maxIter = 200, featuresCol = "features", 
 labelCol = 'label', standardization = True, elasticNetParam = 1)
paramGrid_logreg = ParamGridBuilder().addGrid(logreg.regParam, np.linspace(0.0, 1, 11)).build()
crossval_logreg = CrossValidator(estimator = logreg, 
 estimatorParamMaps = paramGrid_logreg, 
 evaluator = BinaryClassificationEvaluator(), 
 numFolds = 10) 
cvModel_logreg = crossval_logreg.fit(train_df)

Now, the coefficients are below:

cvModel_logreg.bestModel.coefficients

DenseVector([0.0022, -0.0216, -0.0261, -0.0018, 0.0014, -0.0012, -0.0006, 0.0023, -0.0024, -0.0003, -0.0114, 0.0003, 0.0, -0.0018, -0.0163, -0.0009, -0.0022, 0.0009, -0.0017, 0.0005, -0.0024, 0.0006, -0.0025, 0.0015, -0.0025, 0.0007, -0.002, 0.0002, 0.0007, 0.0002, -0.0025, 0.0008, -0.0012, -0.0001, -0.0017, 0.0002, 0.0026, -0.0002, -0.0019, 0.0003, -0.0017, 0.0005, -0.0019, 0.0008, -0.0023, 0.0008, -0.0021, -0.0003, -0.0021, 0.0011, -0.0019, 0.0003, -0.0018, -0.0006, 0.0001, -0.0009, -0.3818, -0.0425, -0.0065, 0.0014, 0.1304, 0.0003, -0.0698, 0.0002, 0.0005, 0.0014, 0.0161, -0.0005, 0.0099, 0.0003, 0.051, -0.0006, -0.0001, -0.0005, 0.3291, -0.0056, 0.0451])

machine-learning pyspark logistic-regression

share|improve this question

edited Mar 26 at 16:58

Fisseha Berhane

asked Mar 26 at 16:15

Fisseha BerhaneFisseha Berhane

80711 gold badge1010 silver badges3030 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  what is optimized value of regParam coming out of cross validation
  
  – abhiieor
  Mar 26 at 17:08
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  cvModel_logreg_lasso4.bestModel._java_obj.getRegParam() gives 0.0
  
  – Fisseha Berhane
  Mar 26 at 17:31
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  that means zero Lasso penalty. If you want lesser number of variables then you must have some penalty there but CV is saying best model is with zero penalty. Analyze CV results and decide.
  
  – abhiieor
  Mar 27 at 8:24
  

  
  
  
  

add a comment | 

1

0

After splitting my data in to training and testing, my training data has about 33 million records. I have 77 features and a binary response. I am fitting a logistic regression with lasso. However, lasso returns non-zero coefficient values for all the columns. I explored the correlation of the features among themselves and some of them have correlation of up to 0.8 and 0.7. Provided that I have many features (77) and the features are correlated, I would expect lasso to filter out at least some of the features. Why is this not happening? I could not share the data because of its size and privacy. But the code I am using is below. I am looking for what could be happening and what to explore.

logreg = LogisticRegression(maxIter = 200, featuresCol = "features", 
 labelCol = 'label', standardization = True, elasticNetParam = 1)
paramGrid_logreg = ParamGridBuilder().addGrid(logreg.regParam, np.linspace(0.0, 1, 11)).build()
crossval_logreg = CrossValidator(estimator = logreg, 
 estimatorParamMaps = paramGrid_logreg, 
 evaluator = BinaryClassificationEvaluator(), 
 numFolds = 10) 
cvModel_logreg = crossval_logreg.fit(train_df)

Now, the coefficients are below:

cvModel_logreg.bestModel.coefficients

DenseVector([0.0022, -0.0216, -0.0261, -0.0018, 0.0014, -0.0012, -0.0006, 0.0023, -0.0024, -0.0003, -0.0114, 0.0003, 0.0, -0.0018, -0.0163, -0.0009, -0.0022, 0.0009, -0.0017, 0.0005, -0.0024, 0.0006, -0.0025, 0.0015, -0.0025, 0.0007, -0.002, 0.0002, 0.0007, 0.0002, -0.0025, 0.0008, -0.0012, -0.0001, -0.0017, 0.0002, 0.0026, -0.0002, -0.0019, 0.0003, -0.0017, 0.0005, -0.0019, 0.0008, -0.0023, 0.0008, -0.0021, -0.0003, -0.0021, 0.0011, -0.0019, 0.0003, -0.0018, -0.0006, 0.0001, -0.0009, -0.3818, -0.0425, -0.0065, 0.0014, 0.1304, 0.0003, -0.0698, 0.0002, 0.0005, 0.0014, 0.0161, -0.0005, 0.0099, 0.0003, 0.051, -0.0006, -0.0001, -0.0005, 0.3291, -0.0056, 0.0451])

machine-learning pyspark logistic-regression

share|improve this question

edited Mar 26 at 16:58

Fisseha Berhane

asked Mar 26 at 16:15

Fisseha BerhaneFisseha Berhane

80711 gold badge1010 silver badges3030 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  what is optimized value of regParam coming out of cross validation
  
  – abhiieor
  Mar 26 at 17:08
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  cvModel_logreg_lasso4.bestModel._java_obj.getRegParam() gives 0.0
  
  – Fisseha Berhane
  Mar 26 at 17:31
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  that means zero Lasso penalty. If you want lesser number of variables then you must have some penalty there but CV is saying best model is with zero penalty. Analyze CV results and decide.
  
  – abhiieor
  Mar 27 at 8:24
  

  
  
  
  

add a comment | 

After splitting my data in to training and testing, my training data has about 33 million records. I have 77 features and a binary response. I am fitting a logistic regression with lasso. However, lasso returns non-zero coefficient values for all the columns. I explored the correlation of the features among themselves and some of them have correlation of up to 0.8 and 0.7. Provided that I have many features (77) and the features are correlated, I would expect lasso to filter out at least some of the features. Why is this not happening? I could not share the data because of its size and privacy. But the code I am using is below. I am looking for what could be happening and what to explore.

logreg = LogisticRegression(maxIter = 200, featuresCol = "features", 
 labelCol = 'label', standardization = True, elasticNetParam = 1)
paramGrid_logreg = ParamGridBuilder().addGrid(logreg.regParam, np.linspace(0.0, 1, 11)).build()
crossval_logreg = CrossValidator(estimator = logreg, 
 estimatorParamMaps = paramGrid_logreg, 
 evaluator = BinaryClassificationEvaluator(), 
 numFolds = 10) 
cvModel_logreg = crossval_logreg.fit(train_df)

Now, the coefficients are below:

cvModel_logreg.bestModel.coefficients

DenseVector([0.0022, -0.0216, -0.0261, -0.0018, 0.0014, -0.0012, -0.0006, 0.0023, -0.0024, -0.0003, -0.0114, 0.0003, 0.0, -0.0018, -0.0163, -0.0009, -0.0022, 0.0009, -0.0017, 0.0005, -0.0024, 0.0006, -0.0025, 0.0015, -0.0025, 0.0007, -0.002, 0.0002, 0.0007, 0.0002, -0.0025, 0.0008, -0.0012, -0.0001, -0.0017, 0.0002, 0.0026, -0.0002, -0.0019, 0.0003, -0.0017, 0.0005, -0.0019, 0.0008, -0.0023, 0.0008, -0.0021, -0.0003, -0.0021, 0.0011, -0.0019, 0.0003, -0.0018, -0.0006, 0.0001, -0.0009, -0.3818, -0.0425, -0.0065, 0.0014, 0.1304, 0.0003, -0.0698, 0.0002, 0.0005, 0.0014, 0.0161, -0.0005, 0.0099, 0.0003, 0.051, -0.0006, -0.0001, -0.0005, 0.3291, -0.0056, 0.0451])

machine-learning pyspark logistic-regression

share|improve this question

edited Mar 26 at 16:58

Fisseha Berhane

asked Mar 26 at 16:15

Fisseha BerhaneFisseha Berhane

80711 gold badge1010 silver badges3030 bronze badges

logreg = LogisticRegression(maxIter = 200, featuresCol = "features", 
 labelCol = 'label', standardization = True, elasticNetParam = 1)
paramGrid_logreg = ParamGridBuilder().addGrid(logreg.regParam, np.linspace(0.0, 1, 11)).build()
crossval_logreg = CrossValidator(estimator = logreg, 
 estimatorParamMaps = paramGrid_logreg, 
 evaluator = BinaryClassificationEvaluator(), 
 numFolds = 10) 
cvModel_logreg = crossval_logreg.fit(train_df)

cvModel_logreg.bestModel.coefficients

share|improve this question

edited Mar 26 at 16:58

Fisseha Berhane

asked Mar 26 at 16:15

Fisseha BerhaneFisseha Berhane

80711 gold badge1010 silver badges3030 bronze badges

share|improve this question

edited Mar 26 at 16:58

Fisseha Berhane

asked Mar 26 at 16:15

Fisseha BerhaneFisseha Berhane

80711 gold badge1010 silver badges3030 bronze badges

asked Mar 26 at 16:15

Fisseha BerhaneFisseha Berhane

80711 gold badge1010 silver badges3030 bronze badges

asked Mar 26 at 16:15

Fisseha BerhaneFisseha Berhane

80711 gold badge1010 silver badges3030 bronze badges