Styjun

If the Moon were impacted by a suitably sized meteor, how long would it take to impact the Earth?

How would a lunar colony attack Earth?

Database Cache Memory in Performance Monitor drops down significantly after DBCC CheckDB

My employer is refusing to give me the pay that was advertised after an internal job move

ULQ2003 not driving a Relay properly

Create two random teams from a list of players

Why didn't General Martok receive discommendation in Star Trek: Deep Space Nine?

Patio gate not at right angle to the house

If I buy and download a game through second Nintendo account do I own it on my main account too?

What do the novel titles of The Expanse series refer to?

What kind of horizontal stabilizer does a Boeing 737 have?

How to identify in Apex what picklist fields are restricted

Should 2FA be enabled on service accounts?

Given mean and SD, can we approximate the underlying distribution?

In the Schrödinger equation, can I have a Hamiltonian without a kinetic term?

Can machine learning learn a function like finding maximum from a list?

Prepare a user to perform an action before proceeding to the next step

How do I make my photos have more impact?

What Marvel character has this 'W' symbol?

Word for soundtrack music which is part of the action of the movie

Should I intervene when a colleague in a different department makes students run laps as part of their grade?

Why would an invisible personal shield be necessary?

Introduction to the Sicilian

How to remove rebar passing through an inaccessible pipe

Restarting program using os.system to save memory?

How to determine CPU and memory consumption from inside a process?Which Python memory profiler is recommended?How to measure actual memory usage of an application or process?How do I discover memory usage of my application in Android?Creating a memory leak with JavaSave plot to image file instead of displaying it using MatplotlibWeb parsing with python beautifulsoup producing inconsistent resultscraping issue (dynamic content)(without selenium)Using Beautiful Soup how can I return this value and use it as an integer?I am getting text error while the code is appicable for on company using python beautifulsoup

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

I'm using Python to scrape a website for specific links or keywords, and want to send a request about every 5 seconds. Initially I was using a while True loop to send my requests every 5 seconds, but with every loop my program used about 1mb of memory more than before. With me wanting to run my program on a Raspberry Pi for potentially multiple days, this would fill up the memory pretty quickly.

Instead of using the while True loop, once my program ran through all the code I use

os.system("python program.py") 

to restart the program, meaning running through the code over and over won't gain any memory and stay at an average of about 38mb.

So my question is: Is there any downside of me doing this? Can any problems occur when I want my computer to run a new instance of a program every 5 seconds for days?

EDIT: added code

import requests
from bs4 import BeautifulSoup, SoupStrainer
import time
import os
import psutil

while True:
 url = "https://en.wikipedia.org/wiki/Main_Page"
 source = requests.get(url).text 
 soup = BeautifulSoup(source, 'lxml') 
 for link in soup.find_all("a"): 
 print(link.get('href'))

 time.sleep(5)
 process = psutil.Process(os.getpid())
 print(process.memory_info().rss)
 time.sleep(5)

python memory while-loop os.system

share|improve this question

edited Mar 26 at 22:32

Viet NaM

asked Mar 26 at 22:02

Viet NaMViet NaM

2555 bronze badges

• 
  
  
  

  
  6
  

  
  
  
  
  
  

  
  
  If the memory consumption is increasing, that means you probably have a memory leak. Instead of restarting the application, fix the memory leak problem.
  
  – zvone
  Mar 26 at 22:04
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  If you are willing to restart your program and just trash any data you've accumulated then it suggests that you're persisting data on each loop that you don't need. Without seeing your code, it's not possible to understand why the memory footprint grows but clearly you don't intend it
  
  – roganjosh
  Mar 26 at 22:04
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This is a classic XY Problem in my opinion.
  
  – Random Davis
  Mar 26 at 22:22
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @zvone I added a code which is essentially what I'm doing, but even this simple code causes my memory to go up every loop. It started with 32mb and gains about 0.7mb every loop.
  
  – Viet NaM
  Mar 26 at 22:31
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  This seems to be a case of premature optimization. There is no memory leak. When I run this code without time.sleep (so memory consumption grows much much faster), it always stays at around 50 MB (as expected). It is normal that it sometimes goes up for various reasons, and it does, but then it comes down again. So everything is fine.
  
  – zvone
  Mar 26 at 23:19
  

  
  
  
  

 | 
show 3 more comments

1

I'm using Python to scrape a website for specific links or keywords, and want to send a request about every 5 seconds. Initially I was using a while True loop to send my requests every 5 seconds, but with every loop my program used about 1mb of memory more than before. With me wanting to run my program on a Raspberry Pi for potentially multiple days, this would fill up the memory pretty quickly.

Instead of using the while True loop, once my program ran through all the code I use

os.system("python program.py") 

to restart the program, meaning running through the code over and over won't gain any memory and stay at an average of about 38mb.

So my question is: Is there any downside of me doing this? Can any problems occur when I want my computer to run a new instance of a program every 5 seconds for days?

EDIT: added code

import requests
from bs4 import BeautifulSoup, SoupStrainer
import time
import os
import psutil

while True:
 url = "https://en.wikipedia.org/wiki/Main_Page"
 source = requests.get(url).text 
 soup = BeautifulSoup(source, 'lxml') 
 for link in soup.find_all("a"): 
 print(link.get('href'))

 time.sleep(5)
 process = psutil.Process(os.getpid())
 print(process.memory_info().rss)
 time.sleep(5)

python memory while-loop os.system

share|improve this question

edited Mar 26 at 22:32

Viet NaM

asked Mar 26 at 22:02

Viet NaMViet NaM

2555 bronze badges

• 
  
  
  

  
  6
  

  
  
  
  
  
  

  
  
  If the memory consumption is increasing, that means you probably have a memory leak. Instead of restarting the application, fix the memory leak problem.
  
  – zvone
  Mar 26 at 22:04
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  If you are willing to restart your program and just trash any data you've accumulated then it suggests that you're persisting data on each loop that you don't need. Without seeing your code, it's not possible to understand why the memory footprint grows but clearly you don't intend it
  
  – roganjosh
  Mar 26 at 22:04
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This is a classic XY Problem in my opinion.
  
  – Random Davis
  Mar 26 at 22:22
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @zvone I added a code which is essentially what I'm doing, but even this simple code causes my memory to go up every loop. It started with 32mb and gains about 0.7mb every loop.
  
  – Viet NaM
  Mar 26 at 22:31
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  This seems to be a case of premature optimization. There is no memory leak. When I run this code without time.sleep (so memory consumption grows much much faster), it always stays at around 50 MB (as expected). It is normal that it sometimes goes up for various reasons, and it does, but then it comes down again. So everything is fine.
  
  – zvone
  Mar 26 at 23:19
  

  
  
  
  

 | 
show 3 more comments

I'm using Python to scrape a website for specific links or keywords, and want to send a request about every 5 seconds. Initially I was using a while True loop to send my requests every 5 seconds, but with every loop my program used about 1mb of memory more than before. With me wanting to run my program on a Raspberry Pi for potentially multiple days, this would fill up the memory pretty quickly.

Instead of using the while True loop, once my program ran through all the code I use

os.system("python program.py") 

to restart the program, meaning running through the code over and over won't gain any memory and stay at an average of about 38mb.

So my question is: Is there any downside of me doing this? Can any problems occur when I want my computer to run a new instance of a program every 5 seconds for days?

EDIT: added code

import requests
from bs4 import BeautifulSoup, SoupStrainer
import time
import os
import psutil

while True:
 url = "https://en.wikipedia.org/wiki/Main_Page"
 source = requests.get(url).text 
 soup = BeautifulSoup(source, 'lxml') 
 for link in soup.find_all("a"): 
 print(link.get('href'))

 time.sleep(5)
 process = psutil.Process(os.getpid())
 print(process.memory_info().rss)
 time.sleep(5)

python memory while-loop os.system

share|improve this question

edited Mar 26 at 22:32

Viet NaM

asked Mar 26 at 22:02

Viet NaMViet NaM

2555 bronze badges

os.system("python program.py") 

import requests
from bs4 import BeautifulSoup, SoupStrainer
import time
import os
import psutil

while True:
 url = "https://en.wikipedia.org/wiki/Main_Page"
 source = requests.get(url).text 
 soup = BeautifulSoup(source, 'lxml') 
 for link in soup.find_all("a"): 
 print(link.get('href'))

 time.sleep(5)
 process = psutil.Process(os.getpid())
 print(process.memory_info().rss)
 time.sleep(5)

share|improve this question

edited Mar 26 at 22:32

Viet NaM

asked Mar 26 at 22:02

Viet NaMViet NaM

2555 bronze badges

share|improve this question

edited Mar 26 at 22:32

Viet NaM

asked Mar 26 at 22:02

Viet NaMViet NaM

2555 bronze badges

asked Mar 26 at 22:02

Viet NaMViet NaM

2555 bronze badges

asked Mar 26 at 22:02

Viet NaMViet NaM

2555 bronze badges

1 Answer
1

active

oldest

votes

1

To be honest it sounds like the program should be re-written. If you are storing data internally which you don't need (which from the sounds of it you are), you need to ask yourself why you are. If you need that data, write it out and reset the variable.

Some more clarity would really help here - ie - the code itself so we could figure out the real problem.

share|improve this answer

answered Mar 26 at 22:10

amoeba_wonderboyamoeba_wonderboy

1122 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I added a code which is essentially what I'm doing, but even this simple code causes my memory to go up every loop. It started with 32mb and gains about 0.7mb every loop.
  
  – Viet NaM
  Mar 26 at 22:32
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Instagram does it. They has some server that runs until the memory consumption is to high, then they restart it. They used a modified version of CPython with the garbage collector disabled to be honest. It wasn't a memory leak like the one in the question.
  
  – Pedro Lacerda
  Mar 26 at 22:34
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55366828%2frestarting-program-using-os-system-to-save-memory%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

To be honest it sounds like the program should be re-written. If you are storing data internally which you don't need (which from the sounds of it you are), you need to ask yourself why you are. If you need that data, write it out and reset the variable.

Some more clarity would really help here - ie - the code itself so we could figure out the real problem.

share|improve this answer

answered Mar 26 at 22:10

amoeba_wonderboyamoeba_wonderboy

1122 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I added a code which is essentially what I'm doing, but even this simple code causes my memory to go up every loop. It started with 32mb and gains about 0.7mb every loop.
  
  – Viet NaM
  Mar 26 at 22:32
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Instagram does it. They has some server that runs until the memory consumption is to high, then they restart it. They used a modified version of CPython with the garbage collector disabled to be honest. It wasn't a memory leak like the one in the question.
  
  – Pedro Lacerda
  Mar 26 at 22:34
  

  
  
  
  

add a comment | 

1

To be honest it sounds like the program should be re-written. If you are storing data internally which you don't need (which from the sounds of it you are), you need to ask yourself why you are. If you need that data, write it out and reset the variable.

Some more clarity would really help here - ie - the code itself so we could figure out the real problem.

share|improve this answer

answered Mar 26 at 22:10

amoeba_wonderboyamoeba_wonderboy

1122 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I added a code which is essentially what I'm doing, but even this simple code causes my memory to go up every loop. It started with 32mb and gains about 0.7mb every loop.
  
  – Viet NaM
  Mar 26 at 22:32
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Instagram does it. They has some server that runs until the memory consumption is to high, then they restart it. They used a modified version of CPython with the garbage collector disabled to be honest. It wasn't a memory leak like the one in the question.
  
  – Pedro Lacerda
  Mar 26 at 22:34
  

  
  
  
  

add a comment | 

To be honest it sounds like the program should be re-written. If you are storing data internally which you don't need (which from the sounds of it you are), you need to ask yourself why you are. If you need that data, write it out and reset the variable.

Some more clarity would really help here - ie - the code itself so we could figure out the real problem.

share|improve this answer

answered Mar 26 at 22:10

amoeba_wonderboyamoeba_wonderboy

1122 bronze badges

share|improve this answer

answered Mar 26 at 22:10

amoeba_wonderboyamoeba_wonderboy

1122 bronze badges

answered Mar 26 at 22:10

amoeba_wonderboyamoeba_wonderboy

1122 bronze badges

answered Mar 26 at 22:10

amoeba_wonderboyamoeba_wonderboy

1122 bronze badges