Styjun

How could Dwarves prevent sand from filling up their settlements

Is a reptile with diamond scales possible?

Who is frowning in the sentence "Daisy looked at Tom frowning"?

Is it possible to view all the attribute data in QGIS

Precedent for disabled Kings

Why were early aviators' trousers flared at the thigh?

Cycling to work - 30 mile return

Reference for electronegativities of different metal oxidation states

Could a chemically propelled craft travel directly between Earth and Mars spaceports?

Bash - Execute two commands and get exit status 1 if first fails

Bash Read: Reading comma separated list, last element is missed

Bash Array of Word-Splitting Headaches

Why does the U.S military use mercenaries?

Very serious stuff - Salesforce bug enabled "Modify All"

Are there any crystals that are theoretically possible, but haven't yet been made?

Better than Rembrandt

Does science define life as "beginning at conception"?

If you attack a Tarrasque while swallowed, what AC do you need to beat to hit it?

How to choose the correct exposure for flower photography?

Is there any official Lore on Keraptis the Wizard, apart from what is in White Plume Mountain?

Would it be possible to set up a franchise in the ancient world?

pwaS eht tirsf dna tasl setterl fo hace dorw

What is the backup for a glass cockpit, if a plane loses power to the displays/controls?

How can I prevent Bash expansion from passing files starting with "-" as argument?

Load and convert a lot of images to array of size (n,224,224,3) from path

Import a module from a relative pathHow to get the filename without the extension from a path in Python?Converting from a string to boolean in Python?Python progression path - From apprentice to guruConverting array to list in JavaExtract file name from path, no matter what the os/path formatConvert list to array in JavaHow to convert 2D float numpy array to 2D int numpy array?Keras load sound instances in batchesLoad images and annotations from CSV and use fit_generator with multi-output models

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

1

I'm currently loading images, creating an array out of it and appending it to a list. Sadly this seems to eat up all of my RAM for the amount of images I'm trying to load (20k).

Code:

def convert_image_to_array(files,relpath):
 images_as_array=[]
 len_files = len(files)
 i = 0
 print("---ConvImg2Arr---")
 print("---STARTING---")
 for file in files:
 images_as_array.append(img_to_array(load_img(relpath+file, target_size=(soll_img_shape, soll_img_shape)))/255)
 if i == int(len_files*0.2):
 print("20% done")
 if i == int(len_files*0.5):
 print("50% done")
 if i == int(len_files*0.8):
 print("80% done")

 i +=1
 print("---DONE---")
 return images_as_array

calling it with X_train being coming from train_test_split:

x_train = convert_image_to_array_opt(X_train,rel_path)

What is an more efficient way to load all those images?

Edit:

Using .flow_from_directory() from Keras solved my issues but I would still like to know how it could be done the way I tried.

python list numpy keras

share|improve this question

edited Mar 23 at 20:22

Phil

asked Mar 23 at 18:41

PhilPhil

1909

add a comment | 

1

I'm currently loading images, creating an array out of it and appending it to a list. Sadly this seems to eat up all of my RAM for the amount of images I'm trying to load (20k).

Code:

def convert_image_to_array(files,relpath):
 images_as_array=[]
 len_files = len(files)
 i = 0
 print("---ConvImg2Arr---")
 print("---STARTING---")
 for file in files:
 images_as_array.append(img_to_array(load_img(relpath+file, target_size=(soll_img_shape, soll_img_shape)))/255)
 if i == int(len_files*0.2):
 print("20% done")
 if i == int(len_files*0.5):
 print("50% done")
 if i == int(len_files*0.8):
 print("80% done")

 i +=1
 print("---DONE---")
 return images_as_array

calling it with X_train being coming from train_test_split:

x_train = convert_image_to_array_opt(X_train,rel_path)

What is an more efficient way to load all those images?

Edit:

Using .flow_from_directory() from Keras solved my issues but I would still like to know how it could be done the way I tried.

python list numpy keras

share|improve this question

edited Mar 23 at 20:22

Phil

asked Mar 23 at 18:41

PhilPhil

1909

add a comment | 

I'm currently loading images, creating an array out of it and appending it to a list. Sadly this seems to eat up all of my RAM for the amount of images I'm trying to load (20k).

Code:

def convert_image_to_array(files,relpath):
 images_as_array=[]
 len_files = len(files)
 i = 0
 print("---ConvImg2Arr---")
 print("---STARTING---")
 for file in files:
 images_as_array.append(img_to_array(load_img(relpath+file, target_size=(soll_img_shape, soll_img_shape)))/255)
 if i == int(len_files*0.2):
 print("20% done")
 if i == int(len_files*0.5):
 print("50% done")
 if i == int(len_files*0.8):
 print("80% done")

 i +=1
 print("---DONE---")
 return images_as_array

calling it with X_train being coming from train_test_split:

x_train = convert_image_to_array_opt(X_train,rel_path)

What is an more efficient way to load all those images?

Edit:

Using .flow_from_directory() from Keras solved my issues but I would still like to know how it could be done the way I tried.

python list numpy keras

share|improve this question

edited Mar 23 at 20:22

Phil

asked Mar 23 at 18:41

PhilPhil

1909

def convert_image_to_array(files,relpath):
 images_as_array=[]
 len_files = len(files)
 i = 0
 print("---ConvImg2Arr---")
 print("---STARTING---")
 for file in files:
 images_as_array.append(img_to_array(load_img(relpath+file, target_size=(soll_img_shape, soll_img_shape)))/255)
 if i == int(len_files*0.2):
 print("20% done")
 if i == int(len_files*0.5):
 print("50% done")
 if i == int(len_files*0.8):
 print("80% done")

 i +=1
 print("---DONE---")
 return images_as_array

x_train = convert_image_to_array_opt(X_train,rel_path)

share|improve this question

edited Mar 23 at 20:22

Phil

asked Mar 23 at 18:41

PhilPhil

1909

share|improve this question

edited Mar 23 at 20:22

Phil

asked Mar 23 at 18:41

PhilPhil

1909

asked Mar 23 at 18:41

PhilPhil

1909

asked Mar 23 at 18:41

PhilPhil

1909

1 Answer
1

active

oldest

votes

0

Assuming that the method load_img is not the bottleneck, convert_image_to_array_opt loads all the images (20k) into memory. However, flow_from_directory methods only loads one bath of images at a time (typical batch sizes are 32, 64, ... 1024)

A possible way to redesign convert_image_to_array_opt will be to take the batch size as argument and load and yield a numpy array loaded with bath_size images only (along with labels). And while training enumerate on convert_image_to_array_opt method which return batch_size X's and y's which you can train on.

share|improve this answer

edited Mar 23 at 22:27

answered Mar 23 at 20:34

mujjigamujjiga

3,59511320

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  load_img is actually from keras.preprocessing.image !
  
  – Phil
  Mar 23 at 22:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  So the assumption that load_img is not a bottleneck is true then. Rest of my answer is the way I normally approach when I write my own data loaders.
  
  – mujjiga
  Mar 23 at 22:26
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55317175%2fload-and-convert-a-lot-of-images-to-array-of-size-n-224-224-3-from-path%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

Assuming that the method load_img is not the bottleneck, convert_image_to_array_opt loads all the images (20k) into memory. However, flow_from_directory methods only loads one bath of images at a time (typical batch sizes are 32, 64, ... 1024)

A possible way to redesign convert_image_to_array_opt will be to take the batch size as argument and load and yield a numpy array loaded with bath_size images only (along with labels). And while training enumerate on convert_image_to_array_opt method which return batch_size X's and y's which you can train on.

share|improve this answer

edited Mar 23 at 22:27

answered Mar 23 at 20:34

mujjigamujjiga

3,59511320

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  load_img is actually from keras.preprocessing.image !
  
  – Phil
  Mar 23 at 22:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  So the assumption that load_img is not a bottleneck is true then. Rest of my answer is the way I normally approach when I write my own data loaders.
  
  – mujjiga
  Mar 23 at 22:26
  

  
  
  
  

add a comment | 

0

Assuming that the method load_img is not the bottleneck, convert_image_to_array_opt loads all the images (20k) into memory. However, flow_from_directory methods only loads one bath of images at a time (typical batch sizes are 32, 64, ... 1024)

A possible way to redesign convert_image_to_array_opt will be to take the batch size as argument and load and yield a numpy array loaded with bath_size images only (along with labels). And while training enumerate on convert_image_to_array_opt method which return batch_size X's and y's which you can train on.

share|improve this answer

edited Mar 23 at 22:27

answered Mar 23 at 20:34

mujjigamujjiga

3,59511320

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  load_img is actually from keras.preprocessing.image !
  
  – Phil
  Mar 23 at 22:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  So the assumption that load_img is not a bottleneck is true then. Rest of my answer is the way I normally approach when I write my own data loaders.
  
  – mujjiga
  Mar 23 at 22:26
  

  
  
  
  

add a comment | 

Assuming that the method load_img is not the bottleneck, convert_image_to_array_opt loads all the images (20k) into memory. However, flow_from_directory methods only loads one bath of images at a time (typical batch sizes are 32, 64, ... 1024)

A possible way to redesign convert_image_to_array_opt will be to take the batch size as argument and load and yield a numpy array loaded with bath_size images only (along with labels). And while training enumerate on convert_image_to_array_opt method which return batch_size X's and y's which you can train on.

share|improve this answer

edited Mar 23 at 22:27

answered Mar 23 at 20:34

mujjigamujjiga

3,59511320

share|improve this answer

edited Mar 23 at 22:27

answered Mar 23 at 20:34

mujjigamujjiga

3,59511320

answered Mar 23 at 20:34

mujjigamujjiga

3,59511320

answered Mar 23 at 20:34

mujjigamujjiga

3,59511320