Styjun

I have a list of product names some of which are redundant or similar:

List = ['CocaCola','CocaCola 3 Oz','Twix','Twix Caramel','Foldgers 3 Oz','Foldgers 10 Oz','Haagen Dazs Caramel','Black Forest Ham','Black Label Whiskey',...] 

I would like write a function that would group similar product names so that it would return a list:

NewList = ['CocaCola','Twix','Foldgers','Haagen Dazs Caramel','Black Forest Ham','Black Label Whiskey',...]

I thought about matching substrings, but that wouldn't work since 'CocaCola 3 Oz' and ''Foldgers 3 Oz' would both map to '3 Oz'.

I also thought about just the first substring in each product name:

 NewList = []
 for w in List: 
 ws = w.split(' ')
 NewList.append(ws[0]) 

But that would map 'Black Forest Ham' and 'Black Label Whiskey' to 'Black'.

How can I get this mapping? I know of beautifulsoup and thought it might help, but I couldn't find any posts that indicate that.

To clarify based on BruceWayne's comments:
I'm getting the list from a Pandas df (don't know why that is relevant?).
'CocaCola' and 'Pepsi' would map to different groups 'CocaCola' and 'Pepsi'. 'Black Forest Ham' and 'Oscar Meyer Ham' would also map to different groups, 'CocaCola' and 'CocaCola Light' would map to the same group 'CocaCola'. Basically I'l looking for grouping based on brand names, not product categories. That what determines similarity.

I already provided an example of what the output would look like based on the input.

I thought beautifulsoup would help because it is a text processing library.

You can achieve what you are trying to do by using clustering algorithms on your dataset.

a = ['CocaCola','CocaCola 3 Oz','Twix','Twix Caramel','Foldgers 3 Oz','Foldgers 10 Oz','Haagen Dazs Caramel','Black Forest Ham','Black Label Whiskey']

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.cluster import KMeans
cv=CountVectorizer()
vect=cv.fit_transform(a)
km=KMeans(n_clusters=6)
km.fit_predict(vect)

OUTPUT:

array([0, 0, 1, 1, 2, 2, 4, 3, 5], dtype=int32)

This tells us that:

Cluster 0: 'CocaCola','CocaCola 3 Oz'

Cluster 1: 'Twix','Twix Caramel'

Cluster 2: 'Foldgers 3 Oz','Foldgers 10 Oz'

Cluster 3: 'Haagen Dazs Caramel'

Cluster 4: 'Black Forest Ham'

Cluster 5: 'Black Label Whiskey'

You first vectorize the your data i.e., you convert each item in your list into 1D array of numbers. I am using a CountVectorizer here (easy to understand and serves the purpose here), but there are other vectorizers available too. Each digit in the 1D array would represent a word and the valueof that digit would represent the number of times it occurs in that text. This link will help you understand better about CountVectorizer aka Bag of Words algorithm.

Once again, there are many clustering algorithms to choose from and I have chosen KMeans Clustering for the same reason as before, easy to understand and implement.This will help you understand KMeans Clustering.

Note: You need to specify the number of clusters you require as mentioned in km=KMeans(n_clusters=6). A change in the value here might change your results. For example,

If km=KMeans(n_clusters=5), 'Black Forest Ham' and 'Black Label Whiskey' will be categorized in the same cluster.

I hope this helps you.

You could use a regular expression to pick up only the name part that's before a number.

products = ['CocaCola','CocaCola 3 Oz','Twix','Twix Caramel','Foldgers 3 Oz','Foldgers 10 Oz','Haagen Dazs Caramel','Black Forest Ham','Black Label Whiskey']
import re
products = list(set(re.findall("(.*?)[0-9]",name+"0")[0].strip() for name in products))
print(products)

# ['Black Label Whiskey', 'CocaCola', 'Black Forest Ham', 'Twix Caramel', 'Twix', 'Haagen Dazs Caramel', 'Foldgers']