Styjun

How to gracefully excuse yourself from a meeting due to emergencies such as a restroom break?

Could flaps be raised upward to serve as spoilers / lift dumpers?

How to get values on a rastar layer to provide me with values

How can a class have multiple methods without breaking the single responsibility principle

How do Canadians get a visa to go to Saudi Arabia?

Delete the following space

A coworker mumbles to herself when working. How can I ask her to stop?

Constant Scan spooling

Were there any unmanned expeditions to the moon that returned to Earth prior to Apollo?

IBM mainframe classic executable file formats

What is the significance of $(logname)?

Can I say "Gesundheit" if someone is coughing?

Difference between HCD and CID collision induced dissociations?

The Enigma Machine (CLI) in Java

Why are sugars in whole fruits not digested the same way sugars in juice are?

How to let cacti grow even if no player is near?

Using Python in a Bash Script

Is this mechanically safe?

Oath of redemption: Does Emmissary of Peace reflect damage taken from Aura of the Guardian?

How can flights operated by the same company have such different prices when marketed by another?

How do discovery writers hibernate?

Can birds evolve without trees?

If the Moon were impacted by a suitably sized meteor, how long would it take to impact the Earth?

Why should I use a big powerstone instead of smaller ones?

Heroku clock process not storing data to a CSV file

How do I write JSON data to a file?“Large data” work flows using pandasImport multiple csv files into pandas and concatenate into one DataFrameHow are Heroku worker and clock processes priced?Celery --beat on Heroku vs Worker and Clock processesHow to use Heroku Worker jobs with custom Clock with PHP programs?Heroku Scheduler vs Heroku Temporize Scheduler, what's the difference?Heroku scheduler vs clock processLoad csv file as a dataframeStoring list into CSV file from webscraping via selenium?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

I am using Heroku to make a webpage that daily scrapes some content from other pages and afterwards shows this on the page. The problem I encountered is that when running the daily Clock.py file that the scraping process is executed but the new CSV file is not stored at all. What do you think is the reason for this?

To provide some further information, the scraper function opens a webpage, scrapes some content and returns a pandas dataframe. What I now want to achieve is to store this dataframe to a folder data/ with the name df_result2.csv. A short remark: The scraping process works perfectly, as I could print the dataframe to the console after each run. The only problem comes from storing this file. Do you have any suggestions what I should change?

#clock.py
from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
import pandas as pd

from components import scraper

sched = BlockingScheduler()

#Example for a job
@sched.scheduled_job('interval', minutes=3)
def timed_job():
 print('This job is run every 3 minutes.')
 result = scraper()
 pd.DataFrame.to_csv(result, "data/df_result2.csv")
 print(result)

python heroku clock

share|improve this question

edited Mar 27 at 2:08

Chris

64.4k1818 gold badges134134 silver badges129129 bronze badges

asked Mar 26 at 22:52

Hoi DuHoi Du

822 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I think what you're looking for is result.to_csv("data/df_result2.csv") since pd.DataFrame.to_csv is an instance bnot static method of the pd.DataFrame class as far as I know. What is the error you're seeing?
  
  – David Waterworth
  Mar 26 at 23:55
  
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @DavidWaterworth Thank you for your reply. Interestingly there is no error message at all. The reason might be due to this euphemeral filesystem as Chris explained. The file may hence not be available to other dynos
  
  – Hoi Du
  Mar 27 at 7:14
  

  
  
  
  

add a comment | 

1

I am using Heroku to make a webpage that daily scrapes some content from other pages and afterwards shows this on the page. The problem I encountered is that when running the daily Clock.py file that the scraping process is executed but the new CSV file is not stored at all. What do you think is the reason for this?

To provide some further information, the scraper function opens a webpage, scrapes some content and returns a pandas dataframe. What I now want to achieve is to store this dataframe to a folder data/ with the name df_result2.csv. A short remark: The scraping process works perfectly, as I could print the dataframe to the console after each run. The only problem comes from storing this file. Do you have any suggestions what I should change?

#clock.py
from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
import pandas as pd

from components import scraper

sched = BlockingScheduler()

#Example for a job
@sched.scheduled_job('interval', minutes=3)
def timed_job():
 print('This job is run every 3 minutes.')
 result = scraper()
 pd.DataFrame.to_csv(result, "data/df_result2.csv")
 print(result)

python heroku clock

share|improve this question

edited Mar 27 at 2:08

Chris

64.4k1818 gold badges134134 silver badges129129 bronze badges

asked Mar 26 at 22:52

Hoi DuHoi Du

822 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I think what you're looking for is result.to_csv("data/df_result2.csv") since pd.DataFrame.to_csv is an instance bnot static method of the pd.DataFrame class as far as I know. What is the error you're seeing?
  
  – David Waterworth
  Mar 26 at 23:55
  
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @DavidWaterworth Thank you for your reply. Interestingly there is no error message at all. The reason might be due to this euphemeral filesystem as Chris explained. The file may hence not be available to other dynos
  
  – Hoi Du
  Mar 27 at 7:14
  

  
  
  
  

add a comment | 

I am using Heroku to make a webpage that daily scrapes some content from other pages and afterwards shows this on the page. The problem I encountered is that when running the daily Clock.py file that the scraping process is executed but the new CSV file is not stored at all. What do you think is the reason for this?

To provide some further information, the scraper function opens a webpage, scrapes some content and returns a pandas dataframe. What I now want to achieve is to store this dataframe to a folder data/ with the name df_result2.csv. A short remark: The scraping process works perfectly, as I could print the dataframe to the console after each run. The only problem comes from storing this file. Do you have any suggestions what I should change?

#clock.py
from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
import pandas as pd

from components import scraper

sched = BlockingScheduler()

#Example for a job
@sched.scheduled_job('interval', minutes=3)
def timed_job():
 print('This job is run every 3 minutes.')
 result = scraper()
 pd.DataFrame.to_csv(result, "data/df_result2.csv")
 print(result)

python heroku clock

share|improve this question

edited Mar 27 at 2:08

Chris

64.4k1818 gold badges134134 silver badges129129 bronze badges

asked Mar 26 at 22:52

Hoi DuHoi Du

822 bronze badges

#clock.py
from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
import pandas as pd

from components import scraper

sched = BlockingScheduler()

#Example for a job
@sched.scheduled_job('interval', minutes=3)
def timed_job():
 print('This job is run every 3 minutes.')
 result = scraper()
 pd.DataFrame.to_csv(result, "data/df_result2.csv")
 print(result)

share|improve this question

edited Mar 27 at 2:08

Chris

64.4k1818 gold badges134134 silver badges129129 bronze badges

asked Mar 26 at 22:52

Hoi DuHoi Du

822 bronze badges

share|improve this question

edited Mar 27 at 2:08

Chris

64.4k1818 gold badges134134 silver badges129129 bronze badges

asked Mar 26 at 22:52

Hoi DuHoi Du

822 bronze badges

edited Mar 27 at 2:08

Chris

64.4k1818 gold badges134134 silver badges129129 bronze badges

edited Mar 27 at 2:08

Chris

64.4k1818 gold badges134134 silver badges129129 bronze badges

asked Mar 26 at 22:52

Hoi DuHoi Du

822 bronze badges

asked Mar 26 at 22:52

Hoi DuHoi Du

822 bronze badges

1 Answer
1

active

oldest

votes

0

The immediate problem is probably that the data/ directory doesn't exist.

But the bigger problem is that Heroku's filesystem is ephemeral. Any changes you make to it will be lost whenever your dyno restarts. This happens frequently and unpredictably (at least once per day).

Since you're generating your data every three minutes the risk is somewhat limited, but there could definitely be times that that file might not exist when you need it.

Instead of storing your data on the filesystem I strongly suggest that you use a proper data store. PostgreSQL works great with Heroku. If you really want to use a file, consider storing it on a third-party service like Amazon S3.

share|improve this answer

answered Mar 27 at 2:06

ChrisChris

64.4k1818 gold badges134134 silver badges129129 bronze badges

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55367324%2fheroku-clock-process-not-storing-data-to-a-csv-file%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

The immediate problem is probably that the data/ directory doesn't exist.

But the bigger problem is that Heroku's filesystem is ephemeral. Any changes you make to it will be lost whenever your dyno restarts. This happens frequently and unpredictably (at least once per day).

Since you're generating your data every three minutes the risk is somewhat limited, but there could definitely be times that that file might not exist when you need it.

Instead of storing your data on the filesystem I strongly suggest that you use a proper data store. PostgreSQL works great with Heroku. If you really want to use a file, consider storing it on a third-party service like Amazon S3.

share|improve this answer

answered Mar 27 at 2:06

ChrisChris

64.4k1818 gold badges134134 silver badges129129 bronze badges

add a comment | 

0

The immediate problem is probably that the data/ directory doesn't exist.

But the bigger problem is that Heroku's filesystem is ephemeral. Any changes you make to it will be lost whenever your dyno restarts. This happens frequently and unpredictably (at least once per day).

Since you're generating your data every three minutes the risk is somewhat limited, but there could definitely be times that that file might not exist when you need it.

Instead of storing your data on the filesystem I strongly suggest that you use a proper data store. PostgreSQL works great with Heroku. If you really want to use a file, consider storing it on a third-party service like Amazon S3.

share|improve this answer

answered Mar 27 at 2:06

ChrisChris

64.4k1818 gold badges134134 silver badges129129 bronze badges

add a comment | 

The immediate problem is probably that the data/ directory doesn't exist.

But the bigger problem is that Heroku's filesystem is ephemeral. Any changes you make to it will be lost whenever your dyno restarts. This happens frequently and unpredictably (at least once per day).

Since you're generating your data every three minutes the risk is somewhat limited, but there could definitely be times that that file might not exist when you need it.

Instead of storing your data on the filesystem I strongly suggest that you use a proper data store. PostgreSQL works great with Heroku. If you really want to use a file, consider storing it on a third-party service like Amazon S3.

share|improve this answer

answered Mar 27 at 2:06

ChrisChris

64.4k1818 gold badges134134 silver badges129129 bronze badges

share|improve this answer

answered Mar 27 at 2:06

ChrisChris

64.4k1818 gold badges134134 silver badges129129 bronze badges

answered Mar 27 at 2:06

ChrisChris

64.4k1818 gold badges134134 silver badges129129 bronze badges

answered Mar 27 at 2:06

ChrisChris

64.4k1818 gold badges134134 silver badges129129 bronze badges