Styjun

Will I be more secure with my own router behind my ISP's router?

What's parked in Mil Moscow helicopter plant?

"Working on a knee"

How can I wire a 9-position switch so that each position turns on one more LED than the one before?

What were wait-states, and why was it only an issue for PCs?

Retract an already submitted Recommendation Letter (written for an undergrad student)

What is the ongoing value of the Kanban board to the developers as opposed to management

Does Prince Arnaud cause someone holding the Princess to lose?

What was Apollo 13's "Little Jolt" after MECO?

Will temporary Dex penalties prevent you from getting the benefits of the "Two Weapon Fighting" feat if your Dex score falls below the prerequisite?

How would you suggest I follow up with coworkers about our deadline that's today?

How would it unbalance gameplay to rule that Weapon Master allows for picking a fighting style?

Raising a bilingual kid. When should we introduce the majority language?

Determinant of a matrix with 2 equal rows

Why I cannot instantiate a class whose constructor is private in a friend class?

What does こした mean?

Is there a verb for listening stealthily?

/bin/ls sorts differently than just ls

Why is water being consumed when my shutoff valve is closed?

What is /etc/mtab in Linux?

Putting Ant-Man on house arrest

TV series episode where humans nuke aliens before decrypting their message that states they come in peace

Variable does not exist: sObjectType (Task.sObjectType)

Is it appropriate to mention a relatable company blog post when you're asked about the company?

Deserializing Spacy results

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

0

I need to run an algorithm on a lot of text files. In order to pre-process them, I use Spacy which has pre-trained models in different languages. Since the pre-processed results are employed in different part of algorithms, it is better to save them on disk once and load them many times. However, the Spacy deserialization method makes an error.
I wrote a simple code to show the error:

de_nlp=spacy.load("de_core_news_sm",disable=['ner', 'parser'])
de_nlp.add_pipe(de_nlp.create_pipe('sentencizer'))
doc = de_nlp(text_file_content)

for ix, sent in enumerate(doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

#Serialization and Deserilization 
doc.to_disk("/tmp/test_result.bin")
new_doc = Doc(Vocab()).from_disk("/tmp/test_result.bin")

for ix, sent in enumerate(new_doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

However, the above example code makes the following error:

Traceback (most recent call last): File "/tmp/test_result.bin", line 14, in <module>
 for ix, sent in enumerate(new_doc.sents, 1): 
File "doc.pyx", line 535, in __get__
ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with: nlp.add_pipe(nlp.create_pipe('sentencizer')) Alternatively, add the dependency parser, or set sentence boundaries by setting doc[i].is_sent_start.

The version of Spacy I use is 2.0.18.

Any information about this topic is appreciated.

python-3.x nlp text-mining spacy natural-language-processing

share|improve this question

edited Mar 22 at 15:38

TrebledJ

4,04821432

asked Mar 22 at 14:43

SahelSoftSahelSoft

3251517

add a comment | 

0

I need to run an algorithm on a lot of text files. In order to pre-process them, I use Spacy which has pre-trained models in different languages. Since the pre-processed results are employed in different part of algorithms, it is better to save them on disk once and load them many times. However, the Spacy deserialization method makes an error.
I wrote a simple code to show the error:

de_nlp=spacy.load("de_core_news_sm",disable=['ner', 'parser'])
de_nlp.add_pipe(de_nlp.create_pipe('sentencizer'))
doc = de_nlp(text_file_content)

for ix, sent in enumerate(doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

#Serialization and Deserilization 
doc.to_disk("/tmp/test_result.bin")
new_doc = Doc(Vocab()).from_disk("/tmp/test_result.bin")

for ix, sent in enumerate(new_doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

However, the above example code makes the following error:

Traceback (most recent call last): File "/tmp/test_result.bin", line 14, in <module>
 for ix, sent in enumerate(new_doc.sents, 1): 
File "doc.pyx", line 535, in __get__
ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with: nlp.add_pipe(nlp.create_pipe('sentencizer')) Alternatively, add the dependency parser, or set sentence boundaries by setting doc[i].is_sent_start.

The version of Spacy I use is 2.0.18.

Any information about this topic is appreciated.

python-3.x nlp text-mining spacy natural-language-processing

share|improve this question

edited Mar 22 at 15:38

TrebledJ

4,04821432

asked Mar 22 at 14:43

SahelSoftSahelSoft

3251517

add a comment | 

I need to run an algorithm on a lot of text files. In order to pre-process them, I use Spacy which has pre-trained models in different languages. Since the pre-processed results are employed in different part of algorithms, it is better to save them on disk once and load them many times. However, the Spacy deserialization method makes an error.
I wrote a simple code to show the error:

de_nlp=spacy.load("de_core_news_sm",disable=['ner', 'parser'])
de_nlp.add_pipe(de_nlp.create_pipe('sentencizer'))
doc = de_nlp(text_file_content)

for ix, sent in enumerate(doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

#Serialization and Deserilization 
doc.to_disk("/tmp/test_result.bin")
new_doc = Doc(Vocab()).from_disk("/tmp/test_result.bin")

for ix, sent in enumerate(new_doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

However, the above example code makes the following error:

Traceback (most recent call last): File "/tmp/test_result.bin", line 14, in <module>
 for ix, sent in enumerate(new_doc.sents, 1): 
File "doc.pyx", line 535, in __get__
ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with: nlp.add_pipe(nlp.create_pipe('sentencizer')) Alternatively, add the dependency parser, or set sentence boundaries by setting doc[i].is_sent_start.

The version of Spacy I use is 2.0.18.

Any information about this topic is appreciated.

python-3.x nlp text-mining spacy natural-language-processing

share|improve this question

edited Mar 22 at 15:38

TrebledJ

4,04821432

asked Mar 22 at 14:43

SahelSoftSahelSoft

3251517

de_nlp=spacy.load("de_core_news_sm",disable=['ner', 'parser'])
de_nlp.add_pipe(de_nlp.create_pipe('sentencizer'))
doc = de_nlp(text_file_content)

for ix, sent in enumerate(doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

#Serialization and Deserilization 
doc.to_disk("/tmp/test_result.bin")
new_doc = Doc(Vocab()).from_disk("/tmp/test_result.bin")

for ix, sent in enumerate(new_doc.sents, 1):
 print("--Sentence number : ".format(ix, sent))
 lemma = [w.lemma_ for w in sent]
 print(f"Lemma ==> lemma")

Traceback (most recent call last): File "/tmp/test_result.bin", line 14, in <module>
 for ix, sent in enumerate(new_doc.sents, 1): 
File "doc.pyx", line 535, in __get__
ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with: nlp.add_pipe(nlp.create_pipe('sentencizer')) Alternatively, add the dependency parser, or set sentence boundaries by setting doc[i].is_sent_start.

share|improve this question

edited Mar 22 at 15:38

TrebledJ

4,04821432

asked Mar 22 at 14:43

SahelSoftSahelSoft

3251517

share|improve this question

edited Mar 22 at 15:38

TrebledJ

4,04821432

asked Mar 22 at 14:43

SahelSoftSahelSoft

3251517

edited Mar 22 at 15:38

TrebledJ

4,04821432

edited Mar 22 at 15:38

TrebledJ

4,04821432

asked Mar 22 at 14:43

SahelSoftSahelSoft

3251517

asked Mar 22 at 14:43

SahelSoftSahelSoft

3251517