Styjun

Highlevel Background

I'm working on a project where in the first step I'm searching for keywords and phrases inside a large text corpus. I want to identify passages/sentences where these keywords occur. Later I want to make these passages accessible through my local postgres db for the user to query information. The data is stored on Azure Blob Storage and I'm using Minio Server to connect my Django application.

Actual Problem

First my shell was killed and after some try-and-error refactoring/debugging a memory error, when running my script that:

1. samples 30 (I want to sample 10000 but it breaks already at low numbers) random text documents from the blob storage,


2. preprocess the raw text for the nlp task,


3. streams the text through spacy's nlp.pipe to get a list of docs and


4. streams the list of docs to PhraseMatcher (which passes on_match the rule_id, start token of a sentence (with match), the sentence, hash_id to a match_list).

At first the shell was killed. I looked into the log files and saw that it was a memory error, but to be honest I'm quite new to this topic.

After rearranging the code I got an MemoryError directly inside the shell. Within the language.pipe() step of streaming the text to spaCy.

Code extracts

Functions

# Function that samples filing_documents
def random_Filings(amount):
 ...
 return random_list

# Function that connects to storage and saves cleaned text
def get_clean_text(random_list):
 try:
 text_contents = S3Client().get_buffer(remote_path)
 ...
return clean_list

# matcher function that performs action on match of PhraseMatcher
def on_match(matcher, doc, id, matches):
 matcher_id, start, end = matches[id]
 rule_id = nlp.vocab.strings[match_id]
 token = doc[start]
 sent_of_token = token.sent
 match_list.append([str(rule_id), sent_of_token.start, sent_of_token, 
 doc.user_data])

def match_text_stream(clean_texts):
 some_pattern = [nlp(text) for text in ('foo', 'bar')]
 some_other_pattern = [nlp(text) for text in ('foo bar', 'barara')]

 matcher = PhraseMAtcher(nlp.vocab)

 matcher.add('SOME', on_match, *some_pattern)
 matcher.add('OTHER', on_match, *some_other_pattern)

 doc_list = []

 for doc in nlp.pipe(list_of_text, barch_size=30):
 doc_list.append(doc)

 for doc in matcher.pipi(doc_list, batch_size=30):
 pass

Problemsteps:

match_list = []

nlp = en_core_web_sm.load()
sample_list = random_Filings(30)
clean_texts = get_clean_text(sample_list)
match_text_stream(clean_text)

print(match_list)

Error Message

MemoryError
<string> in in match_text_stream(clean_text)

../spacy/language.py in pipe(self, texts, as_tubles, n thready, batch_size, disable cleanup, component_cfg)

709 origingal_strings_data = None
710 nr_seen = 0
711 for doc in docs:
712 yield doc
713 if cleanup:

...

MemoryError


../tick/neural/_classes/convolution.py in begin_update(self, X__bi, drop)

31
32 def(bedin_update(self,X__bi, drop=0.0):
33 X__bo = self.ops.seqcol(X__bi, self.nW)
34 finish_update = self._get_finsih_update()
35 return X__bo, finish_update

ops.pyx in thinc.neural.ops.NumpyOps.seq2col()
ops.pyx in thinc.neural.ops.NumpyOps.allocate()

MemoryError:

The solution is to cut your documents up into smaller pieces before training. Paragraph units work quite well, or possibly sections.