Styjun

Suing a Police Officer Instead of the Police Department

Can gravitational waves pass through a black hole?

How would it unbalance gameplay to rule that Weapon Master allows for picking a fighting style?

When speaking, how do you change your mind mid-sentence?

What is the purpose of the side handle on a hand ("eggbeater") drill?

What is the numbering system used for the DSN dishes?

/bin/ls sorts differently than just ls

How did Elite on the NES work?

Determinant of a matrix with 2 equal rows

Is Bran literally the world's memory?

Marquee sign letters

Does using the Inspiration rules for character defects encourage My Guy Syndrome?

France's Public Holidays' Puzzle

What is a 'Key' in computer science?

How would you suggest I follow up with coworkers about our deadline that's today?

Why do people think Winterfell crypts is the safest place for women, children & old people?

Retract an already submitted Recommendation Letter (written for an undergrad student)

Is it appropriate to mention a relatable company blog post when you're asked about the company?

to see a doctor

What were wait-states, and why was it only an issue for PCs?

What is the definining line between a helicopter and a drone a person can ride in?

Why aren't road bicycle wheels tiny?

Getting AggregateResult variables from Execute Anonymous Window

What's parked in Mil Moscow helicopter plant?

How to run my webscraper with a Flask REST API without waiting for it to finish before returning a response? [duplicate]

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

0

This question already has an answer here:

• 
  How to use threading in Python?
  
  17 answers
  
  


• 
  Running background Celery task in Flask
  
  1 answer
  
  

I have a RESTFUL API that I built with Flask which runs a webscraper to download files.

I want to host it on a linux EC2 instance and serve it using NGINX and Gunicorn.

I have been testing my API with postman but because the scraper takes about 10 minutes to finish postman hangs waiting for a response.

My flask app looks something like this:

from flask import Flask

application = Flask(__name__)

@application.route('/scraper/run', methods=['POST'])
def init_scrape():
 data = request.json
 command = './web_scrape.py -us "0" -p "1" -url "2"'.format(data['username'], data['password'], data['url'])

 # This takes about 10 minutes
 output = subprocess.check_output(['bash','-c', command])

 return jsonify('Scraping this site: ': request.json["url"]), 201

if __name__ == '__main__':
 application.run(host="0.0.0.0", port="8080")

Is there a way I can run my scraper without having to wait for it to finish before returning some data to postman?

python flask

share|improve this question

asked Mar 22 at 14:53

Connor McCannConnor McCann

12510

marked as duplicate by davidism flask
Users with the  flask badge can single-handedly close flask questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Mar 22 at 15:04

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

add a comment | 

0

This question already has an answer here:

• 
  How to use threading in Python?
  
  17 answers
  
  


• 
  Running background Celery task in Flask
  
  1 answer
  
  

I have a RESTFUL API that I built with Flask which runs a webscraper to download files.

I want to host it on a linux EC2 instance and serve it using NGINX and Gunicorn.

I have been testing my API with postman but because the scraper takes about 10 minutes to finish postman hangs waiting for a response.

My flask app looks something like this:

from flask import Flask

application = Flask(__name__)

@application.route('/scraper/run', methods=['POST'])
def init_scrape():
 data = request.json
 command = './web_scrape.py -us "0" -p "1" -url "2"'.format(data['username'], data['password'], data['url'])

 # This takes about 10 minutes
 output = subprocess.check_output(['bash','-c', command])

 return jsonify('Scraping this site: ': request.json["url"]), 201

if __name__ == '__main__':
 application.run(host="0.0.0.0", port="8080")

Is there a way I can run my scraper without having to wait for it to finish before returning some data to postman?

python flask

share|improve this question

asked Mar 22 at 14:53

Connor McCannConnor McCann

12510

marked as duplicate by davidism flask
Users with the  flask badge can single-handedly close flask questions as duplicates and reopen them as needed.

StackExchange.ready(function()
if (StackExchange.options.isMobile) return;

$('.dupe-hammer-message-hover:not(.hover-bound)').each(function()
var $hover = $(this).addClass('hover-bound'),
$msg = $hover.siblings('.dupe-hammer-message');

$hover.hover(
function()
$hover.showInfoMessage('',
messageElement: $msg.clone().show(),
transient: false,
position: my: 'bottom left', at: 'top center', offsetTop: -7 ,
dismissable: false,
relativeToBody: true
);
,
function()
StackExchange.helpers.removeMessages();

);
);
);
Mar 22 at 15:04

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

add a comment | 

This question already has an answer here:

• 
  How to use threading in Python?
  
  17 answers
  
  


• 
  Running background Celery task in Flask
  
  1 answer
  
  

I have a RESTFUL API that I built with Flask which runs a webscraper to download files.

I want to host it on a linux EC2 instance and serve it using NGINX and Gunicorn.

I have been testing my API with postman but because the scraper takes about 10 minutes to finish postman hangs waiting for a response.

My flask app looks something like this:

from flask import Flask

application = Flask(__name__)

@application.route('/scraper/run', methods=['POST'])
def init_scrape():
 data = request.json
 command = './web_scrape.py -us "0" -p "1" -url "2"'.format(data['username'], data['password'], data['url'])

 # This takes about 10 minutes
 output = subprocess.check_output(['bash','-c', command])

 return jsonify('Scraping this site: ': request.json["url"]), 201

if __name__ == '__main__':
 application.run(host="0.0.0.0", port="8080")

Is there a way I can run my scraper without having to wait for it to finish before returning some data to postman?

python flask

share|improve this question

asked Mar 22 at 14:53

Connor McCannConnor McCann

12510

from flask import Flask

application = Flask(__name__)

@application.route('/scraper/run', methods=['POST'])
def init_scrape():
 data = request.json
 command = './web_scrape.py -us "0" -p "1" -url "2"'.format(data['username'], data['password'], data['url'])

 # This takes about 10 minutes
 output = subprocess.check_output(['bash','-c', command])

 return jsonify('Scraping this site: ': request.json["url"]), 201

if __name__ == '__main__':
 application.run(host="0.0.0.0", port="8080")

share|improve this question

asked Mar 22 at 14:53

Connor McCannConnor McCann

12510

share|improve this question

asked Mar 22 at 14:53

Connor McCannConnor McCann

12510

asked Mar 22 at 14:53

Connor McCannConnor McCann

12510

asked Mar 22 at 14:53

Connor McCannConnor McCann

12510