Styjun

Convergence of series of normally distributed random variables

What are the occurences of total war in the Native Americans?

How to gently end involvement with an online community?

How does the OS tell whether an "Address is already in use"?

Expanding powers of expressions of the form ax+b

Should I stick with American terminology in my English set young adult book?

Can you grapple/shove when affected by the Crown of Madness spell?

How many lines of code does the original TeX contain?

What is a natural problem in theory of computation?

Cooking Scrambled Eggs

Why did my folder names end up like this, and how can I fix this using a script?

Make utility using LINQ

using resizegather inside tabular: how to remove extra vertical space above and below the equation?

Is it legal for source code containing undefined behavior to crash the compiler?

Why is getting a PhD considered "financially irresponsible"?

I don't have the theoretical background in my PhD topic. I can't justify getting the degree

Can an ISO file damage—or infect—the machine it's being burned on?

How do you capitalize agile costs with less mature teams?

Why is strlen so complex in C?

How would a low-tech device be able to alert its user?

What happened to the HDEV ISS Experiment? Is it over?

Is first Ubuntu user root?

What to look for in a spotting scope?

How long do you think advanced cybernetic implants would plausibly last?

How to encode when writing to file?

How to merge two dictionaries in a single expression?How do I check if a list is empty?How do I check whether a file exists without exceptions?Encode URL in JavaScript?How do I sort a dictionary by value?Twitter image encoding challengeHow do I list all files of a directory?How to read a file line-by-line into a list?Writing Unicode text to a text file?How do I lowercase a string in Python?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

I am trying to write some Data to a file. In some instances, obviously depending on the Data I am trying to write, I get a UnicodeEncodeError (UnicodeEncodeError: 'charmap' codec can't encode character 'U0001f622' in position 141: character maps to )
I did some research and found out that I can encode the data I am writing with the encode function.

This is the code prior to modifying it (not supporting Unicode):

 scriptDir = os.path.dirname(__file__)
 path = os.path.join(scriptDir, filename)
 with open(path, 'w') as fp:
 for sentence in iobTriplets:
 fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))
 fp.write("n")
 fp.write("n")

So I though maybe I could just add the encoding when writing like that:

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]).encode('utf8') for triplet in sentence))

But that doesn't work as I am getting the following error:
TypeError: sequence item 0: expected str instance, bytes found

I also tried opening the file in byte mode with adding a b behind the w. However that didn't yield any results.

Does anybody know how to fix this?
Btw: I am using python 3.

python unicode encoding

share|improve this question

edited Mar 27 at 19:49

Malonga

asked Mar 27 at 19:40

MalongaMalonga

2055 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Please post the original error. Also, I assume this is Python 3?
  
  – juanpa.arrivillaga
  Mar 27 at 19:42
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  the mode should be encode('utf-8')
  
  – C.Nivs
  Mar 27 at 19:43
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Probably, yours system has a default encoding that is being picked up by open, maybe something like ASCII. So, try using open(path, 'w', encoding='utf-8')
  
  – juanpa.arrivillaga
  Mar 27 at 19:44
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @C.Nivs either works there. In fact, if it's Python 3, then you can just do encode(). I am curious as to why not opening the file in binary mode isn't working. What does "I also tried opening the file in byte mode with adding a b behind the w. However that didn't yield any results." mean exactly
  
  – juanpa.arrivillaga
  Mar 27 at 19:45
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Anyway, you need to use b'n'.join(...) if you are going to be joining bytes. that is likely the source of your error, but then you will have to use binary mode when opening the file
  
  – juanpa.arrivillaga
  Mar 27 at 19:46
  
  

  
  
  
  

 | 
show 1 more comment

0

I am trying to write some Data to a file. In some instances, obviously depending on the Data I am trying to write, I get a UnicodeEncodeError (UnicodeEncodeError: 'charmap' codec can't encode character 'U0001f622' in position 141: character maps to )
I did some research and found out that I can encode the data I am writing with the encode function.

This is the code prior to modifying it (not supporting Unicode):

 scriptDir = os.path.dirname(__file__)
 path = os.path.join(scriptDir, filename)
 with open(path, 'w') as fp:
 for sentence in iobTriplets:
 fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))
 fp.write("n")
 fp.write("n")

So I though maybe I could just add the encoding when writing like that:

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]).encode('utf8') for triplet in sentence))

But that doesn't work as I am getting the following error:
TypeError: sequence item 0: expected str instance, bytes found

I also tried opening the file in byte mode with adding a b behind the w. However that didn't yield any results.

Does anybody know how to fix this?
Btw: I am using python 3.

python unicode encoding

share|improve this question

edited Mar 27 at 19:49

Malonga

asked Mar 27 at 19:40

MalongaMalonga

2055 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Please post the original error. Also, I assume this is Python 3?
  
  – juanpa.arrivillaga
  Mar 27 at 19:42
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  the mode should be encode('utf-8')
  
  – C.Nivs
  Mar 27 at 19:43
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Probably, yours system has a default encoding that is being picked up by open, maybe something like ASCII. So, try using open(path, 'w', encoding='utf-8')
  
  – juanpa.arrivillaga
  Mar 27 at 19:44
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @C.Nivs either works there. In fact, if it's Python 3, then you can just do encode(). I am curious as to why not opening the file in binary mode isn't working. What does "I also tried opening the file in byte mode with adding a b behind the w. However that didn't yield any results." mean exactly
  
  – juanpa.arrivillaga
  Mar 27 at 19:45
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  Anyway, you need to use b'n'.join(...) if you are going to be joining bytes. that is likely the source of your error, but then you will have to use binary mode when opening the file
  
  – juanpa.arrivillaga
  Mar 27 at 19:46
  
  

  
  
  
  

 | 
show 1 more comment

I am trying to write some Data to a file. In some instances, obviously depending on the Data I am trying to write, I get a UnicodeEncodeError (UnicodeEncodeError: 'charmap' codec can't encode character 'U0001f622' in position 141: character maps to )
I did some research and found out that I can encode the data I am writing with the encode function.

This is the code prior to modifying it (not supporting Unicode):

 scriptDir = os.path.dirname(__file__)
 path = os.path.join(scriptDir, filename)
 with open(path, 'w') as fp:
 for sentence in iobTriplets:
 fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))
 fp.write("n")
 fp.write("n")

So I though maybe I could just add the encoding when writing like that:

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]).encode('utf8') for triplet in sentence))

But that doesn't work as I am getting the following error:
TypeError: sequence item 0: expected str instance, bytes found

I also tried opening the file in byte mode with adding a b behind the w. However that didn't yield any results.

Does anybody know how to fix this?
Btw: I am using python 3.

python unicode encoding

share|improve this question

edited Mar 27 at 19:49

Malonga

asked Mar 27 at 19:40

MalongaMalonga

2055 bronze badges

 scriptDir = os.path.dirname(__file__)
 path = os.path.join(scriptDir, filename)
 with open(path, 'w') as fp:
 for sentence in iobTriplets:
 fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))
 fp.write("n")
 fp.write("n")

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]).encode('utf8') for triplet in sentence))

share|improve this question

edited Mar 27 at 19:49

Malonga

asked Mar 27 at 19:40

MalongaMalonga

2055 bronze badges

share|improve this question

edited Mar 27 at 19:49

Malonga

asked Mar 27 at 19:40

MalongaMalonga

2055 bronze badges

asked Mar 27 at 19:40

MalongaMalonga

2055 bronze badges

asked Mar 27 at 19:40

MalongaMalonga

2055 bronze badges

1 Answer
1

active

oldest

votes

1

You have already opened the file with automatic encoding. There is no need to manually encode anything unless you are writing to binary.

You can specify any supported encoding in open():

 with open(path, 'w', encoding='utf-16be') as fp:

Unless the file is opened as binary, you need to remove the str.encode() in the fp.write():

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))

share|improve this answer

edited Mar 27 at 19:56

answered Mar 27 at 19:49

mossymountainmossymountain

10366 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks man! That also works
  
  – Malonga
  Mar 27 at 21:21
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55385259%2fhow-to-encode-when-writing-to-file%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

You have already opened the file with automatic encoding. There is no need to manually encode anything unless you are writing to binary.

You can specify any supported encoding in open():

 with open(path, 'w', encoding='utf-16be') as fp:

Unless the file is opened as binary, you need to remove the str.encode() in the fp.write():

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))

share|improve this answer

edited Mar 27 at 19:56

answered Mar 27 at 19:49

mossymountainmossymountain

10366 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks man! That also works
  
  – Malonga
  Mar 27 at 21:21
  

  
  
  
  

add a comment | 

1

You have already opened the file with automatic encoding. There is no need to manually encode anything unless you are writing to binary.

You can specify any supported encoding in open():

 with open(path, 'w', encoding='utf-16be') as fp:

Unless the file is opened as binary, you need to remove the str.encode() in the fp.write():

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))

share|improve this answer

edited Mar 27 at 19:56

answered Mar 27 at 19:49

mossymountainmossymountain

10366 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks man! That also works
  
  – Malonga
  Mar 27 at 21:21
  

  
  
  
  

add a comment | 

You have already opened the file with automatic encoding. There is no need to manually encode anything unless you are writing to binary.

You can specify any supported encoding in open():

 with open(path, 'w', encoding='utf-16be') as fp:

Unless the file is opened as binary, you need to remove the str.encode() in the fp.write():

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))

share|improve this answer

edited Mar 27 at 19:56

answered Mar 27 at 19:49

mossymountainmossymountain

10366 bronze badges

 with open(path, 'w', encoding='utf-16be') as fp:

fp.write("n".join(" ".format(triplet[0],triplet[1],triplet[2]) for triplet in sentence))

share|improve this answer

edited Mar 27 at 19:56

answered Mar 27 at 19:49

mossymountainmossymountain

10366 bronze badges

answered Mar 27 at 19:49

mossymountainmossymountain

10366 bronze badges

answered Mar 27 at 19:49

mossymountainmossymountain

10366 bronze badges