Styjun

Why would Basel III prevent price discovery at credit markets?

Meaning of " brothers in arms"

Why is destructor not called in operator delete?

numpy 1D array: mask elements that repeat more than n times

Right way to say I disagree with the design but ok I will do

Did Bercow say he would have sent the EU extension-request letter himself, had Johnson not done so?

Why buy a first class ticket on Southern trains?

Uncountably many functions coinciding only finitely many values

Continents with simplex noise

C function to check the validity of a date in DD.MM.YYYY format

What is the difference between Anführer and Führer?

Has Turkey released the Greek soldiers they apprehended in 2018?

Is Dom based XSS still a valid security concern in modern browsers?

Echo bracket symbol to terminal

Double feature: Weightier

Replacing triangulated categories with something better

Why is the past tense of vomit generally spelled 'vomited' rather than 'vomitted'?

How offensive is Fachidiot?

Is velocity a valid measure of team and process improvement?

Run "cd" command as superuser in Linux

Precious Stone, as Clear as Diamond

Uncooked peppers and garlic in olive oil fizzled when opened

How can I determine if two vertices on a polygon are consecutive?

AniPop - The anime downloader

Estimator choice for mapping string independent variable to string categorical dependent variable

NLP software for classification of large datasetsMethods for automated synonym detectionEstimating dependent variable as sum of functions of independent variablesHow to apply machine learning to fuzzy matchingPython - How to intuit word from abbreviated text using NLP?What is the difference between “feature” and “independent variable”?What's the ideal way to include dictionaries (gazetteer) in spaCy to improve NER?Does linear regression work with a categorical independent variable & continuous dependent variable?relation in between a categorical dependent variable and combination of independent variablesHow to compare the continuous dependent variable with categorical independent variable?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;

0

I'm attempting to build a predictive model that can map text-based vendor-provided descriptions of a service to around 800 standardized service codes, based on a training set of about 13,000 correctly mapped services.

Each standardized service code also has a standardized description, which is usually similar to the vendor-provided description (i.e, some of the words used are the same), but not identical. Descriptions are typically 3-10 word in length

My main issue is that I'm not sure what type of estimator will be appropriate for this problem.

I have tried using simple fuzzy matching approaches, including:

• Counting matching words/characters between the vendor-provided and standardized service descriptions and selecting the one with the most matches


• Trying to find the standardized service description with the minimum Levenshtein distance

These have not worked particularly well due to the use of synonymous but different word choices within the vendor-provided and standardized descriptions.

I have also considered using a decision tree, but it seems infeasible given 800+ possible outcomes.

Which type of estimator can I use to solve this problem?

machine-learning nlp data-science

share|improve this question

asked Mar 28 at 21:35

J-SharpJ-Sharp

1

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You can first use a summarization technique to extract the most useful words from the description and then use a word embedding technique like word2vec to calculate the similarity between the summary words and the service codes choosing the service code which is most similar. Simple word distance techniques like Levenshtein distance won't work well here because of synonyms
  
  – Ken4scholars
  Mar 29 at 8:37
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks for that @Ken4scholars. Are there any specific summarization techniques that you can recommend?
  
  – J-Sharp
  Mar 29 at 14:20
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You can do an extractive summarization using TF-IDF to calculate the relevance of each word in the description and rank them, then select the highest ranked word
  
  – Ken4scholars
  Mar 29 at 16:50
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Got it, thank you very much @Ken4scholars
  
  – J-Sharp
  Apr 1 at 18:07
  

  
  
  
  

add a comment
 | 

0

I'm attempting to build a predictive model that can map text-based vendor-provided descriptions of a service to around 800 standardized service codes, based on a training set of about 13,000 correctly mapped services.

Each standardized service code also has a standardized description, which is usually similar to the vendor-provided description (i.e, some of the words used are the same), but not identical. Descriptions are typically 3-10 word in length

My main issue is that I'm not sure what type of estimator will be appropriate for this problem.

I have tried using simple fuzzy matching approaches, including:

• Counting matching words/characters between the vendor-provided and standardized service descriptions and selecting the one with the most matches


• Trying to find the standardized service description with the minimum Levenshtein distance

These have not worked particularly well due to the use of synonymous but different word choices within the vendor-provided and standardized descriptions.

I have also considered using a decision tree, but it seems infeasible given 800+ possible outcomes.

Which type of estimator can I use to solve this problem?

machine-learning nlp data-science

share|improve this question

asked Mar 28 at 21:35

J-SharpJ-Sharp

1

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You can first use a summarization technique to extract the most useful words from the description and then use a word embedding technique like word2vec to calculate the similarity between the summary words and the service codes choosing the service code which is most similar. Simple word distance techniques like Levenshtein distance won't work well here because of synonyms
  
  – Ken4scholars
  Mar 29 at 8:37
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks for that @Ken4scholars. Are there any specific summarization techniques that you can recommend?
  
  – J-Sharp
  Mar 29 at 14:20
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  You can do an extractive summarization using TF-IDF to calculate the relevance of each word in the description and rank them, then select the highest ranked word
  
  – Ken4scholars
  Mar 29 at 16:50
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Got it, thank you very much @Ken4scholars
  
  – J-Sharp
  Apr 1 at 18:07
  

  
  
  
  

add a comment
 | 

I'm attempting to build a predictive model that can map text-based vendor-provided descriptions of a service to around 800 standardized service codes, based on a training set of about 13,000 correctly mapped services.

Each standardized service code also has a standardized description, which is usually similar to the vendor-provided description (i.e, some of the words used are the same), but not identical. Descriptions are typically 3-10 word in length

My main issue is that I'm not sure what type of estimator will be appropriate for this problem.

I have tried using simple fuzzy matching approaches, including:

• Counting matching words/characters between the vendor-provided and standardized service descriptions and selecting the one with the most matches


• Trying to find the standardized service description with the minimum Levenshtein distance

These have not worked particularly well due to the use of synonymous but different word choices within the vendor-provided and standardized descriptions.

I have also considered using a decision tree, but it seems infeasible given 800+ possible outcomes.

Which type of estimator can I use to solve this problem?

machine-learning nlp data-science

share|improve this question

asked Mar 28 at 21:35

J-SharpJ-Sharp

1

share|improve this question

asked Mar 28 at 21:35

J-SharpJ-Sharp

1

share|improve this question

asked Mar 28 at 21:35

J-SharpJ-Sharp

1

asked Mar 28 at 21:35

J-SharpJ-Sharp

1

asked Mar 28 at 21:35

J-SharpJ-Sharp

1