Styjun

Can a user sell my software (MIT license) without modification?

Avoiding cliches when writing gods

Question about JavaScript Math.random() and basic logic

Does the "6 seconds per round" rule apply to speaking/roleplaying during combat situations?

When writing an error prompt, should we end the sentence with a exclamation mark or a dot?

Is the term 'open source' a trademark?

Notation of last measure of a song with a pickup measure

Translating 'Liber'

How many times can you cast a card exiled by Release to the Wind?

How bad would a partial hash leak be, realistically?

How to generate random points without duplication?

What can cause the front wheel to lock up when going over a small bump?

What's the right way to purge recursively with apt?

Quickest way to find characteristic polynomial from a given matrix

Average spam confidence

What is the advantage of carrying a tripod and ND-filters when you could use image stacking instead?

2.8 is missing the Carve option in the Boolean Modifier

Is it possible to (7 day) schedule sleep time of a hard drive?

How to translate “Me doing X” like in online posts?

Is it recommended against to open-source the code of a webapp?

How is it possible that Gollum speaks Westron?

Subtables with equal width?

Secure offsite backup, even in the case of hacker root access

What's up with this leaf?

Improving performance of complex logical conditions on numpy arrays

Fastest way to determine if an integer's square root is an integerImprove INSERT-per-second performance of SQLite?How to print the full NumPy array, without truncation?Peak detection in a 2D arrayDump a NumPy array into a csv file“Large data” work flows using pandasOptimizing logical operations on nested numpy arraysSwift Beta performance: sorting arraysReplacing a 32-bit loop counter with 64-bit introduces crazy performance deviationsnumpy array access in condition extremely slower than condition with direct value

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

1

I need to evaluate many logical conditions on a large 2D "NUMPY" array, and collect the overall result in a boolean "RESULT" numpy array.

A simple example where all conditions are linked with an AND statement could be:

RESULT= cond1(NUMPY) & cond2(NUMPY) & cond3(NUMPY) & ....

I would like to understand if there is a way to optimize performance.

For example in this case, if the first condition (cond1) is False for most of the values in the NUMPY array it will be a waste of resources evaluating all other conditions on those values since the AND conditions will anyway generate a False in the final RESULT array.

Any ideas?

python performance numpy optimization logic

share|improve this question

asked Mar 24 at 15:32

PaoloPaolo

204

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Python and and or short circuit, but only for scalar conditions. With whole numpy whole-array operations, each condition is evaluated, and then the values are combined. You'd have to use numba or cython to construct a faster iterative test that implements short-circuiting.
  
  – hpaulj
  Mar 24 at 15:38
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thank you for the explanation and suggestion, I am not so familiar with numba and cython yet, but I will look into those if I do not find another way :)
  
  – Paolo
  Mar 24 at 19:48
  

  
  
  
  

add a comment | 

1

I need to evaluate many logical conditions on a large 2D "NUMPY" array, and collect the overall result in a boolean "RESULT" numpy array.

A simple example where all conditions are linked with an AND statement could be:

RESULT= cond1(NUMPY) & cond2(NUMPY) & cond3(NUMPY) & ....

I would like to understand if there is a way to optimize performance.

For example in this case, if the first condition (cond1) is False for most of the values in the NUMPY array it will be a waste of resources evaluating all other conditions on those values since the AND conditions will anyway generate a False in the final RESULT array.

Any ideas?

python performance numpy optimization logic

share|improve this question

asked Mar 24 at 15:32

PaoloPaolo

204

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Python and and or short circuit, but only for scalar conditions. With whole numpy whole-array operations, each condition is evaluated, and then the values are combined. You'd have to use numba or cython to construct a faster iterative test that implements short-circuiting.
  
  – hpaulj
  Mar 24 at 15:38
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thank you for the explanation and suggestion, I am not so familiar with numba and cython yet, but I will look into those if I do not find another way :)
  
  – Paolo
  Mar 24 at 19:48
  

  
  
  
  

add a comment | 

I need to evaluate many logical conditions on a large 2D "NUMPY" array, and collect the overall result in a boolean "RESULT" numpy array.

A simple example where all conditions are linked with an AND statement could be:

RESULT= cond1(NUMPY) & cond2(NUMPY) & cond3(NUMPY) & ....

I would like to understand if there is a way to optimize performance.

For example in this case, if the first condition (cond1) is False for most of the values in the NUMPY array it will be a waste of resources evaluating all other conditions on those values since the AND conditions will anyway generate a False in the final RESULT array.

Any ideas?

python performance numpy optimization logic

share|improve this question

asked Mar 24 at 15:32

PaoloPaolo

204

share|improve this question

asked Mar 24 at 15:32

PaoloPaolo

204

share|improve this question

asked Mar 24 at 15:32

PaoloPaolo

204

asked Mar 24 at 15:32

PaoloPaolo

204

asked Mar 24 at 15:32

PaoloPaolo

204

1 Answer
1

active

oldest

votes

1

You can do the short-circuiting by hand, though I should add that this is probably only worth it in rather extreme cases.

Here is an example of 99 chained logical ands. The short circuiting is done either using the where keyword or using fancy indexing. The second but not the first gives a decent speed up for this example.

import numpy as np

a = np.random.random((1000,))*1.5
c = np.random.random((100, 1))*1.5

def direct():
 return ((a+c) < np.arccos(np.cos(a+c)*0.99)).all(0)

def trickya():
 out = np.ones(a.shape, '?')
 for ci in c:
 np.logical_and(out, np.less(np.add(a, ci, where=out), np.arccos(np.multiply(np.cos(np.add(a, ci, where=out), where=out), 0.99, where=out), where=out), where=out), out=out, where=out)
 return out

def trickyb():
 idx, = np.where((a+c[0]) < np.arccos(np.cos(a+c[0])*0.99))
 for ci in c[1:]:
 idx = idx[(a[idx]+ci) < np.arccos(np.cos(a[idx]+ci)*0.99)]
 out = np.zeros(a.shape, '?')
 out[idx] = True
 return out

assert (direct()==trickya()).all()
assert (direct()==trickyb()).all()

from timeit import timeit

print('direct ', timeit(direct, number=100))
print('where kw', timeit(trickya, number=100))
print('indexing', timeit(trickyb, number=100))

Sample run:

direct 0.49512664100620896
where kw 0.494946873979643
indexing 0.17760096595156938

share|improve this answer

answered Mar 24 at 18:21

Paul PanzerPaul Panzer

32.6k21846

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Very interesting! I will try and have a look if I can use a similar way to speed up my code :). Thank you!
  
  – Paolo
  Mar 24 at 19:50
  
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55325419%2fimproving-performance-of-complex-logical-conditions-on-numpy-arrays%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

You can do the short-circuiting by hand, though I should add that this is probably only worth it in rather extreme cases.

Here is an example of 99 chained logical ands. The short circuiting is done either using the where keyword or using fancy indexing. The second but not the first gives a decent speed up for this example.

import numpy as np

a = np.random.random((1000,))*1.5
c = np.random.random((100, 1))*1.5

def direct():
 return ((a+c) < np.arccos(np.cos(a+c)*0.99)).all(0)

def trickya():
 out = np.ones(a.shape, '?')
 for ci in c:
 np.logical_and(out, np.less(np.add(a, ci, where=out), np.arccos(np.multiply(np.cos(np.add(a, ci, where=out), where=out), 0.99, where=out), where=out), where=out), out=out, where=out)
 return out

def trickyb():
 idx, = np.where((a+c[0]) < np.arccos(np.cos(a+c[0])*0.99))
 for ci in c[1:]:
 idx = idx[(a[idx]+ci) < np.arccos(np.cos(a[idx]+ci)*0.99)]
 out = np.zeros(a.shape, '?')
 out[idx] = True
 return out

assert (direct()==trickya()).all()
assert (direct()==trickyb()).all()

from timeit import timeit

print('direct ', timeit(direct, number=100))
print('where kw', timeit(trickya, number=100))
print('indexing', timeit(trickyb, number=100))

Sample run:

direct 0.49512664100620896
where kw 0.494946873979643
indexing 0.17760096595156938

share|improve this answer

answered Mar 24 at 18:21

Paul PanzerPaul Panzer

32.6k21846

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Very interesting! I will try and have a look if I can use a similar way to speed up my code :). Thank you!
  
  – Paolo
  Mar 24 at 19:50
  
  

  
  
  
  

add a comment | 

1

You can do the short-circuiting by hand, though I should add that this is probably only worth it in rather extreme cases.

Here is an example of 99 chained logical ands. The short circuiting is done either using the where keyword or using fancy indexing. The second but not the first gives a decent speed up for this example.

import numpy as np

a = np.random.random((1000,))*1.5
c = np.random.random((100, 1))*1.5

def direct():
 return ((a+c) < np.arccos(np.cos(a+c)*0.99)).all(0)

def trickya():
 out = np.ones(a.shape, '?')
 for ci in c:
 np.logical_and(out, np.less(np.add(a, ci, where=out), np.arccos(np.multiply(np.cos(np.add(a, ci, where=out), where=out), 0.99, where=out), where=out), where=out), out=out, where=out)
 return out

def trickyb():
 idx, = np.where((a+c[0]) < np.arccos(np.cos(a+c[0])*0.99))
 for ci in c[1:]:
 idx = idx[(a[idx]+ci) < np.arccos(np.cos(a[idx]+ci)*0.99)]
 out = np.zeros(a.shape, '?')
 out[idx] = True
 return out

assert (direct()==trickya()).all()
assert (direct()==trickyb()).all()

from timeit import timeit

print('direct ', timeit(direct, number=100))
print('where kw', timeit(trickya, number=100))
print('indexing', timeit(trickyb, number=100))

Sample run:

direct 0.49512664100620896
where kw 0.494946873979643
indexing 0.17760096595156938

share|improve this answer

answered Mar 24 at 18:21

Paul PanzerPaul Panzer

32.6k21846

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Very interesting! I will try and have a look if I can use a similar way to speed up my code :). Thank you!
  
  – Paolo
  Mar 24 at 19:50
  
  

  
  
  
  

add a comment | 

You can do the short-circuiting by hand, though I should add that this is probably only worth it in rather extreme cases.

Here is an example of 99 chained logical ands. The short circuiting is done either using the where keyword or using fancy indexing. The second but not the first gives a decent speed up for this example.

import numpy as np

a = np.random.random((1000,))*1.5
c = np.random.random((100, 1))*1.5

def direct():
 return ((a+c) < np.arccos(np.cos(a+c)*0.99)).all(0)

def trickya():
 out = np.ones(a.shape, '?')
 for ci in c:
 np.logical_and(out, np.less(np.add(a, ci, where=out), np.arccos(np.multiply(np.cos(np.add(a, ci, where=out), where=out), 0.99, where=out), where=out), where=out), out=out, where=out)
 return out

def trickyb():
 idx, = np.where((a+c[0]) < np.arccos(np.cos(a+c[0])*0.99))
 for ci in c[1:]:
 idx = idx[(a[idx]+ci) < np.arccos(np.cos(a[idx]+ci)*0.99)]
 out = np.zeros(a.shape, '?')
 out[idx] = True
 return out

assert (direct()==trickya()).all()
assert (direct()==trickyb()).all()

from timeit import timeit

print('direct ', timeit(direct, number=100))
print('where kw', timeit(trickya, number=100))
print('indexing', timeit(trickyb, number=100))

Sample run:

direct 0.49512664100620896
where kw 0.494946873979643
indexing 0.17760096595156938

share|improve this answer

answered Mar 24 at 18:21

Paul PanzerPaul Panzer

32.6k21846

import numpy as np

a = np.random.random((1000,))*1.5
c = np.random.random((100, 1))*1.5

def direct():
 return ((a+c) < np.arccos(np.cos(a+c)*0.99)).all(0)

def trickya():
 out = np.ones(a.shape, '?')
 for ci in c:
 np.logical_and(out, np.less(np.add(a, ci, where=out), np.arccos(np.multiply(np.cos(np.add(a, ci, where=out), where=out), 0.99, where=out), where=out), where=out), out=out, where=out)
 return out

def trickyb():
 idx, = np.where((a+c[0]) < np.arccos(np.cos(a+c[0])*0.99))
 for ci in c[1:]:
 idx = idx[(a[idx]+ci) < np.arccos(np.cos(a[idx]+ci)*0.99)]
 out = np.zeros(a.shape, '?')
 out[idx] = True
 return out

assert (direct()==trickya()).all()
assert (direct()==trickyb()).all()

from timeit import timeit

print('direct ', timeit(direct, number=100))
print('where kw', timeit(trickya, number=100))
print('indexing', timeit(trickyb, number=100))

direct 0.49512664100620896
where kw 0.494946873979643
indexing 0.17760096595156938

share|improve this answer

answered Mar 24 at 18:21

Paul PanzerPaul Panzer

32.6k21846

answered Mar 24 at 18:21

Paul PanzerPaul Panzer

32.6k21846

answered Mar 24 at 18:21

Paul PanzerPaul Panzer

32.6k21846