Styjun

I'm really not sure where to post this question, as this is more of a data governance question than a programming one, and is also somewhat subjective. I will remove/repost elsewhere if this is not the right place.

I am setting up a glossary of terms using a bottom-up approach. Namely, I am initially generating the terms from all of the unique logical column names from different databases and aggregating them into one central location, the glossary. The clean-up and creation of relationships is proving to be an exceptionally difficult and manual process so far. This particular post concerns automating the creation of taxonomies/hierarchical groupings within the larger list of glossary terms.

Let's say that I have a list of ten terms in the glossary below. The terms in my glossary which I know to be related are not necessarily united by any naming convention, so I'm simulating that with this list:

['term a',
 'second term',
 'term 3',
 'term d',
 'term number five',
 'sixth term',
 'seventh term',
 'term eight',
 'ninth term',
 'term 10']

I know for a fact that sixth term is a parent term of term a, term number five and term 10, and ideally I would want to create that association in something like a Python dictionary of lists, as follows:

'sixth term': ['term a', 'term number five', 'term 10']

So basically, my problem is that:

• I have a series of terms that I know to be related in a hierarchical fashion


• The related terms do not necessarily follow a standard naming convention

My initial thoughts on how to deal with this are:

• Trying to use string-matching on the definitions of these terms


• Finding any common terminology that does exist between the related terms and starting from there


• Using a library like difflib to do sequence-matching on terms and their definitions

The problem with using definitions as inputs is that the definitions are not always so good and trying to detect semantic similarity might not get useful results. And I am skeptical about doing a sequence match since, as I mentioned, the related terms do not necessarily follow the same naming convention and a sequence match might not give good results.

If necessary, I can just create the associations manually, but since the glossary is so large (about 9,000 terms) I really want to avoid that. I also suspect that there is a way to do this with machine learning as this seems like a classification problem, but I'll be honest in saying that I am really a novice in ML and really don't know what kind of algorithm would be able to do what I'm seeking.

Basically, I have a problem that is somewhat unique in that it's a governance problem that has spilled over into a programming one, and right now I can only think of creating these associations manually. Not looking for code as much as I am for libraries, tools, or general suggestions on how to deal with a kind of problem like this.