Styjun

Must I use my personal social media account for work?

Is tuition reimbursement a good idea if you have to stay with the job

Can an open source licence be revoked if it violates employer's IP?

A life of PhD: is it feasible?

Keeping track of theme when improvising

Is it possible to have battery technology that can't be duplicated?

Can I get a photo of an Ancient Arrow?

How (un)safe is it to ride barefoot?

What do I need to do, tax-wise, for a sudden windfall?

Is fission/fusion to iron the most efficient way to convert mass to energy?

Why would a car salesman tell me not to get my credit pulled again?

Is it advisable to add a location heads-up when a scene changes in a novel?

Placement of positioning lights on A320 winglets

Realistic, logical way for men with medieval-era weaponry to compete with much larger and physically stronger foes

Why are ambiguous grammars bad?

Purpose of cylindrical attachments on Power Transmission towers

Is plausible to have subspecies with & without separate sexes?

Simple log rotation script

Was the Lonely Mountain, where Smaug lived, a volcano?

Boss making me feel guilty for leaving the company at the end of my internship

Is Jesus the last Prophet?

Is there a radar system monitoring the UK mainland border?

As easy as Three, Two, One... How fast can you go from Five to Four?

Jam with honey & without pectin has a saucy consistency always

Script performs better when run through PyCharm than when run directly in command line

PyCharm performance better than CMD python commandPyCharm cannot catch breakpoints if the python code is running in Maya's Script EditorErrors when running code from command line which are not in PyCharmRunning Python scripts through the Windows Command LinePyQt only gives SIGSEGV when running via PyCharmTheano uses gpu from the command line, but not from PyCharmPyCharm: Executing the script in the console rather than 'run'PYTHONPATH in PyCharm and Windows 10 Command LineRunning python scripts through PyCharm TerminalHow to Run Python script that contains xlwings package in Command PromptImportError when running dev_appserver.py from pycharm

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

2

That very same question was already asked here but got no answers.

I wrote a script that takes very long to complete, and am using a module (tqdm) to give me feedback on its progress. When run on a remote environment through PyCharm, the script performance is between 800 and 1000 iterations per second, which yields an estimated total run time somewhere between 4 and 5 hours.

Problems arise when I try to run the script through a classic Linux command line, on the same remote environment. Under the same circumstances, within the same environment, using the same command, with the only difference that I typed it myself instead of having PyCharm issue it for me, the script performance drops to 200-300 iterations per second, resulting in an estimated 10-16 hours run.

In PyCharm, the run config window for my script looks like this:

As you can see, the used interpreter is located on my development server, in the virtual environment used by the whole project.

I had to hide the script name and parameters, which you may find inconvenient, but I don't believe my issue is related to the script itself or its contents anyway.

In order to run the same script through the command line, here is what I do:

• SSH to my server


• 
  cd to my project folder


• activate my environment


• export PYTHONPATH=$(pwd)


• run the same script with the same options

...which should actually result in the same behaviour, right? Well yes, but not at the same speed. There is this 70% performance drop that happens for reasons that I can't figure out, and which seems to be documented nowhere on the Internet.

I've also tried, in an attempt to perfectly mock what Pycharm does:

• copying the exact command output by PyCharm upon running the script, and pasting it in my terminal (to no avail)


• issuing export PYTHONUNBUFFERED=1 before running the script, which I knew would change nothing (it didn't)

I'm losing my mind over this. Is this the result of some shady PyCharm shenanigans?

Any kind of input related to anything PyCharm does which could even remotely speed up some aspect of a Python script execution is appreciated!

Thanks in advance.

python pycharm

share|improve this question

asked Mar 25 at 0:23

deqyradeqyra

3351415

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Open pycharm File> Setting > Project Interpreter. At top there would be a 'Project Interpreter' heading and in front of it would be an address like C/users....... Change it to python located in appdata folder. Run the program and check runtime. This isn't the answer, this is just a test which would lead to answer.
  
  – Obaid Ur Rehman
  Mar 25 at 4:21
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @ObaidUrRehman unfortunately I'm not going to be able to do that... Before giving you feedback on the estimated runtime, a pre-processing step needs to take place (fetching all of a DB's contents and trim it to keep only the useful data). That requires about 95GB of memory and my personal laptop would have a seizure if I were to try and run it locally :( What metric were you trying to get?
  
  – deqyra
  Mar 25 at 9:25
  

  
  
  
  

add a comment | 

2

That very same question was already asked here but got no answers.

I wrote a script that takes very long to complete, and am using a module (tqdm) to give me feedback on its progress. When run on a remote environment through PyCharm, the script performance is between 800 and 1000 iterations per second, which yields an estimated total run time somewhere between 4 and 5 hours.

Problems arise when I try to run the script through a classic Linux command line, on the same remote environment. Under the same circumstances, within the same environment, using the same command, with the only difference that I typed it myself instead of having PyCharm issue it for me, the script performance drops to 200-300 iterations per second, resulting in an estimated 10-16 hours run.

In PyCharm, the run config window for my script looks like this:

As you can see, the used interpreter is located on my development server, in the virtual environment used by the whole project.

I had to hide the script name and parameters, which you may find inconvenient, but I don't believe my issue is related to the script itself or its contents anyway.

In order to run the same script through the command line, here is what I do:

• SSH to my server


• 
  cd to my project folder


• activate my environment


• export PYTHONPATH=$(pwd)


• run the same script with the same options

...which should actually result in the same behaviour, right? Well yes, but not at the same speed. There is this 70% performance drop that happens for reasons that I can't figure out, and which seems to be documented nowhere on the Internet.

I've also tried, in an attempt to perfectly mock what Pycharm does:

• copying the exact command output by PyCharm upon running the script, and pasting it in my terminal (to no avail)


• issuing export PYTHONUNBUFFERED=1 before running the script, which I knew would change nothing (it didn't)

I'm losing my mind over this. Is this the result of some shady PyCharm shenanigans?

Any kind of input related to anything PyCharm does which could even remotely speed up some aspect of a Python script execution is appreciated!

Thanks in advance.

python pycharm

share|improve this question

asked Mar 25 at 0:23

deqyradeqyra

3351415

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Open pycharm File> Setting > Project Interpreter. At top there would be a 'Project Interpreter' heading and in front of it would be an address like C/users....... Change it to python located in appdata folder. Run the program and check runtime. This isn't the answer, this is just a test which would lead to answer.
  
  – Obaid Ur Rehman
  Mar 25 at 4:21
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @ObaidUrRehman unfortunately I'm not going to be able to do that... Before giving you feedback on the estimated runtime, a pre-processing step needs to take place (fetching all of a DB's contents and trim it to keep only the useful data). That requires about 95GB of memory and my personal laptop would have a seizure if I were to try and run it locally :( What metric were you trying to get?
  
  – deqyra
  Mar 25 at 9:25
  

  
  
  
  

add a comment | 

That very same question was already asked here but got no answers.

I wrote a script that takes very long to complete, and am using a module (tqdm) to give me feedback on its progress. When run on a remote environment through PyCharm, the script performance is between 800 and 1000 iterations per second, which yields an estimated total run time somewhere between 4 and 5 hours.

Problems arise when I try to run the script through a classic Linux command line, on the same remote environment. Under the same circumstances, within the same environment, using the same command, with the only difference that I typed it myself instead of having PyCharm issue it for me, the script performance drops to 200-300 iterations per second, resulting in an estimated 10-16 hours run.

In PyCharm, the run config window for my script looks like this:

As you can see, the used interpreter is located on my development server, in the virtual environment used by the whole project.

I had to hide the script name and parameters, which you may find inconvenient, but I don't believe my issue is related to the script itself or its contents anyway.

In order to run the same script through the command line, here is what I do:

• SSH to my server


• 
  cd to my project folder


• activate my environment


• export PYTHONPATH=$(pwd)


• run the same script with the same options

...which should actually result in the same behaviour, right? Well yes, but not at the same speed. There is this 70% performance drop that happens for reasons that I can't figure out, and which seems to be documented nowhere on the Internet.

I've also tried, in an attempt to perfectly mock what Pycharm does:

• copying the exact command output by PyCharm upon running the script, and pasting it in my terminal (to no avail)


• issuing export PYTHONUNBUFFERED=1 before running the script, which I knew would change nothing (it didn't)

I'm losing my mind over this. Is this the result of some shady PyCharm shenanigans?

Any kind of input related to anything PyCharm does which could even remotely speed up some aspect of a Python script execution is appreciated!

Thanks in advance.

python pycharm

share|improve this question

asked Mar 25 at 0:23

deqyradeqyra

3351415

share|improve this question

asked Mar 25 at 0:23

deqyradeqyra

3351415

share|improve this question

asked Mar 25 at 0:23

deqyradeqyra

3351415

asked Mar 25 at 0:23

deqyradeqyra

3351415

asked Mar 25 at 0:23

deqyradeqyra

3351415