Styjun

Is there a way to work with iterators instead of (for example) numpy.ndarray in numpy?

For example, imagine I have a 2D-array and I want to know if there is a row that only contain even numbers:

import numpy as np

x = np.array([[1, 2], [2, 4], [3, 6]])
np.any(np.all(x % 2 == 0, axis=1))

Is there a way to do this kind of things without instantiating the intermediate objects in memory? (or maybe it is already the case and I just don't know it) In this example, that would mean having an iterator over [False True False] instead of an array. In other words, can we do something that would be equivalent to:

has_an_even_row = False 
for row in x:
 if np.all(row % 2 == 0):
 has_an_even_row = True
 break

My question doesn't only concern all and any but all function/methods in numpy. If it isn't possible I wonder if there is a practical reason for not having this in numpy. (Maybe everyone thinks it's useless, that would be a good reason)

The number of temporary arrays may be more than you realize:

In [224]: x = np.array([[1, 2], [2, 4], [3, 6]]) 
In [225]: x % 2 
Out[225]: 
array([[1, 0],
 [0, 0],
 [1, 0]])
In [226]: _ == 0 
Out[226]: 
array([[False, True],
 [ True, True],
 [False, True]])
In [227]: np.all(_, axis=1) 
Out[227]: array([False, True, False])
In [228]: np.any(_) 
Out[228]: True

In this case, working row by row would save on calculating the last row's values.

The last any step might short-circuit, stopping when it hits the True - that's an implementation detail.

A thoroughly iterative, no excess calculations method would be something like:

In [231]: val = False 
 ...: for row in x: 
 ...: for col in row: 
 ...: if col%2!=0: 
 ...: break 
 ...: val=(row,col) 
 ...: break 

In [232]: val 
Out[232]: (array([2, 4]), 2)

This approach would make sense if I were writing in C or a lisp like language, where testing, memory management, and calculations all occur at the same code level. But it wouldn't be very modular or reusable.

The idea underlying numpy is to provide a comprehensive set of compiled building blocks. Those blocks won't be optimal for all tasks, but on the whole they are fast and easy to use.

It's generally recommended to use the given building blocks for quick development. Once it's working then worry about improving the speed of time critical steps.

The numpy library doesn't give you very many tools to use some of the conventional Python protocols because it is focused on performance within a narrow domain (numeric computation). The whole purpose of numpy is to do numeric operations that are slow in pure Python much more quickly (close to your hardware's maximum speed, like code written in a lower level language like C) without loosing all of the benefits of Python (like garbage collection and easy to read syntax).

The downside to focusing on a narrow domain is that you lose some benefits of more general code. So your for loop code can do less work than numpy does, because it can short-circuit, breaking out of the iteration as soon as the result is known. It doesn't need to do the modulus for every row if it found the result it needs already.

But I suspect if you test it, your numpy code may still going to be faster a lot of the time (test on real data, not trivial stuff like in your example)! Even though it computes a whole bunch of intermediate results up front, the low level operations are so much faster than the equivalent in pure Python that it doesn't matter that it has to iterate over the whole array.