Styjun

How do I remove this inheritance-related code smell?

How did Gollum enter Moria?

In the US, can a former president run again?

Why is it easier to balance a non-moving bike standing up than sitting down?

Where should a runway for a spaceplane be located?

How much steel armor can you wear and still be able to swim?

Non-misogynistic way to say “asshole”?

Covering index used despite missing column

How do I see debug logs for Change Data Capture triggers in Salesforce?

How do internally carried IR missiles acquire a lock?

What triggered jesuits' ban on infinitesimals in 1632?

Print one file per line using echo

Prisoner on alien planet escapes by making up a story about ghost companions and wins the war

I just entered the USA without passport control at Atlanta airport

Why does independence imply zero correlation?

Explicit song lyrics checker

Is there a term for the belief that "if it's legal, it's moral"?

Does a proton have a binding energy?

How many people are necessary to maintain modern civilisation?

Cut the gold chain

What are the current battlegrounds for people’s “rights” in the UK?

Justifying Affordable Bespoke Spaceships

What mathematical theory is required for high frequency trading?

What is "industrial ethernet"?

How to map a dataset of filenames to a dataset of file contents

Keras2 ImageDataGenerator or TensorFlow tf.data?What does tf.nn.conv2d do in tensorflow?Tensorflow Dataset map: input seems to be a placeholder which causes an error in tf.read_fileHow to use Dataset API to read TFRecords file of lists of variant length?Tensorflow dataset data preprocessing is done once for the whole dataset or for each call to iterator.next()?Difference between `Dataset.from_tensors` and `Dataset.from_tensor_slices`?Splitting dataset element into multiple dataset elementsEvaluating the tensors within the mapping function of tf.data.Dataset.map?Loading a NumPy array into a TensorHow do I get and use value from a tensor within a TF 2.0 Dataset map step?Convert a variable sized numpy array to Tensorflow Tensors

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

0

1

For example, I have a tensorflow dataset where each element is a tf.string Tensor represents a filename of an image file. Now I want to map this filename dataset to a dataset of image content Tensors.

I wrote code like this, but it doesn't work because map function can't execute eagerly. (Raises an error saying Tensor type has no attribute named numpy.)

def parseline(line):
 filename = line.numpy()
 image = some_library.open_image(filename).to_numpy()
 return image

dataset = dataset.map(parseline)

tensorflow

share|improve this question

asked Mar 25 at 6:46

lucky yanglucky yang

114110

add a comment | 

0

1

For example, I have a tensorflow dataset where each element is a tf.string Tensor represents a filename of an image file. Now I want to map this filename dataset to a dataset of image content Tensors.

I wrote code like this, but it doesn't work because map function can't execute eagerly. (Raises an error saying Tensor type has no attribute named numpy.)

def parseline(line):
 filename = line.numpy()
 image = some_library.open_image(filename).to_numpy()
 return image

dataset = dataset.map(parseline)

tensorflow

share|improve this question

asked Mar 25 at 6:46

lucky yanglucky yang

114110

add a comment | 

For example, I have a tensorflow dataset where each element is a tf.string Tensor represents a filename of an image file. Now I want to map this filename dataset to a dataset of image content Tensors.

I wrote code like this, but it doesn't work because map function can't execute eagerly. (Raises an error saying Tensor type has no attribute named numpy.)

def parseline(line):
 filename = line.numpy()
 image = some_library.open_image(filename).to_numpy()
 return image

dataset = dataset.map(parseline)

tensorflow

share|improve this question

asked Mar 25 at 6:46

lucky yanglucky yang

114110

def parseline(line):
 filename = line.numpy()
 image = some_library.open_image(filename).to_numpy()
 return image

dataset = dataset.map(parseline)

share|improve this question

asked Mar 25 at 6:46

lucky yanglucky yang

114110

share|improve this question

asked Mar 25 at 6:46

lucky yanglucky yang

114110

asked Mar 25 at 6:46

lucky yanglucky yang

114110

asked Mar 25 at 6:46

lucky yanglucky yang

114110

1 Answer
1

active

oldest

votes

1

Basically, it can be done the following way:

path = 'path_to_images'

files = [os.path.join(path, i) for i in os.listdir(path)] # If you need to create a list of filenames, because tf functions require tensors

def parse_image(filename):
 file = tf.io.read_file(filename) # this will work only with filename as tensor
 image = tf.image.decode_image(f)
 return img

dataset = tf.data.Dataset.from_tensor_slices(files)
dataset = dataset.map(parse_image).batch(1)

if you're in eager mode just iterate over dataset

 for i in dataset: 
 print(i)

If not, you'll need an iterator

iterator = dataset.make_one_shot_iterator()
with tf.Session as sess:
 sess.run(iterator.get_next())

share|improve this answer

answered Mar 25 at 13:00

SharkySharky

2,5542919

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Yes, there is tf.image.decode_image. But what if I want to deal with audio files or any other files that have to be processed with external libraries?
  
  – lucky yang
  Mar 26 at 9:33
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you can try tensorflow.org/api_docs/python/tf/io, in some cases i guess decode_raw will do. there used to be tf.contrib.ffmpeg.encode_audio but it's deprecated. But there was some alternative method of dealing with such cases
  
  – Sharky
  Mar 26 at 14:14
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55332476%2fhow-to-map-a-dataset-of-filenames-to-a-dataset-of-file-contents%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

Basically, it can be done the following way:

path = 'path_to_images'

files = [os.path.join(path, i) for i in os.listdir(path)] # If you need to create a list of filenames, because tf functions require tensors

def parse_image(filename):
 file = tf.io.read_file(filename) # this will work only with filename as tensor
 image = tf.image.decode_image(f)
 return img

dataset = tf.data.Dataset.from_tensor_slices(files)
dataset = dataset.map(parse_image).batch(1)

if you're in eager mode just iterate over dataset

 for i in dataset: 
 print(i)

If not, you'll need an iterator

iterator = dataset.make_one_shot_iterator()
with tf.Session as sess:
 sess.run(iterator.get_next())

share|improve this answer

answered Mar 25 at 13:00

SharkySharky

2,5542919

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Yes, there is tf.image.decode_image. But what if I want to deal with audio files or any other files that have to be processed with external libraries?
  
  – lucky yang
  Mar 26 at 9:33
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you can try tensorflow.org/api_docs/python/tf/io, in some cases i guess decode_raw will do. there used to be tf.contrib.ffmpeg.encode_audio but it's deprecated. But there was some alternative method of dealing with such cases
  
  – Sharky
  Mar 26 at 14:14
  

  
  
  
  

add a comment | 

1

Basically, it can be done the following way:

path = 'path_to_images'

files = [os.path.join(path, i) for i in os.listdir(path)] # If you need to create a list of filenames, because tf functions require tensors

def parse_image(filename):
 file = tf.io.read_file(filename) # this will work only with filename as tensor
 image = tf.image.decode_image(f)
 return img

dataset = tf.data.Dataset.from_tensor_slices(files)
dataset = dataset.map(parse_image).batch(1)

if you're in eager mode just iterate over dataset

 for i in dataset: 
 print(i)

If not, you'll need an iterator

iterator = dataset.make_one_shot_iterator()
with tf.Session as sess:
 sess.run(iterator.get_next())

share|improve this answer

answered Mar 25 at 13:00

SharkySharky

2,5542919

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Yes, there is tf.image.decode_image. But what if I want to deal with audio files or any other files that have to be processed with external libraries?
  
  – lucky yang
  Mar 26 at 9:33
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you can try tensorflow.org/api_docs/python/tf/io, in some cases i guess decode_raw will do. there used to be tf.contrib.ffmpeg.encode_audio but it's deprecated. But there was some alternative method of dealing with such cases
  
  – Sharky
  Mar 26 at 14:14
  

  
  
  
  

add a comment | 

Basically, it can be done the following way:

path = 'path_to_images'

files = [os.path.join(path, i) for i in os.listdir(path)] # If you need to create a list of filenames, because tf functions require tensors

def parse_image(filename):
 file = tf.io.read_file(filename) # this will work only with filename as tensor
 image = tf.image.decode_image(f)
 return img

dataset = tf.data.Dataset.from_tensor_slices(files)
dataset = dataset.map(parse_image).batch(1)

if you're in eager mode just iterate over dataset

 for i in dataset: 
 print(i)

If not, you'll need an iterator

iterator = dataset.make_one_shot_iterator()
with tf.Session as sess:
 sess.run(iterator.get_next())

share|improve this answer

answered Mar 25 at 13:00

SharkySharky

2,5542919

path = 'path_to_images'

files = [os.path.join(path, i) for i in os.listdir(path)] # If you need to create a list of filenames, because tf functions require tensors

def parse_image(filename):
 file = tf.io.read_file(filename) # this will work only with filename as tensor
 image = tf.image.decode_image(f)
 return img

dataset = tf.data.Dataset.from_tensor_slices(files)
dataset = dataset.map(parse_image).batch(1)

 for i in dataset: 
 print(i)

iterator = dataset.make_one_shot_iterator()
with tf.Session as sess:
 sess.run(iterator.get_next())

share|improve this answer

answered Mar 25 at 13:00

SharkySharky

2,5542919

answered Mar 25 at 13:00

SharkySharky

2,5542919

answered Mar 25 at 13:00

SharkySharky

2,5542919