Styjun

Is there a method for differentiating informative comments from commented out code?

Users forgotting to regenerate PDF before sending it

Why do people prefer metropolitan areas, considering monsters and villains?

Moving millions of files to a different directory with specfic name patterns

Draw a diagram with rectangles

Did depressed people far more accurately estimate how many monsters they killed in a video game?

Other Space Shuttle O-ring failures

How do I separate enchants from items?

Redirecting stderr using exec

I'm feeling like my character doesn't fit the campaign

Tesco's Burger Relish Best Before End date number

What's the adjective used to qualify a software that's available on your first boot?

US citizen traveling with Peruvian passport

Is it possible to complete a PhD in CS in 3 years?

Is it possible for a character at any level to cast all 44 Cantrips in one week without Magic Items?

Clarinets in the Rite of Spring

Write a function

How do ballistic trajectories work in a ring world?

What factors could lead to bishops establishing monastic armies?

How to understand flavors and when to use combination of them?

Where are the Wazirs?

As a supervisor, what feedback would you expect from a PhD who quits?

How to evaluate the performance of open source solver?

Four ships at the ocean with the same distance

improve performance of python django models.py query

Calling an external command in PythonWhat are metaclasses in Python?Finding the index of an item given a list containing it in PythonWhat is the difference between Python's list methods append and extend?How can I safely create a nested directory?Does Python have a ternary conditional operator?How to get the current time in PythonHow can I make a time delay in Python?Does Python have a string 'contains' substring method?Why is “1000000000000000 in range(1000000000000001)” so fast in Python 3?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

0

I am loading the website data into Postgres using Python-Django, there are around 9 csv files that i am loading to database but its taking so long ~10 hrs to fetch the data from website, and load it to Postgres. I wanted to increase the performance of the query. Can you guys please help with that?

This is just 1 dataframe, but i have 9 more similar to that, overall data is less than 500k records

from django.db import models 
from django_pandas.managers import DataFrameManager 
import pandas as pd 
from sqlalchemy import create_engine 
import zipfile 
import os 
from urllib.request import urlopen 
import urllib.request 
import io 
from io import BytesIO 

class mf(models.Model):
 pg_engine = create_engine('postgresql://user:password@server:host/db')
 zf = zipfile.ZipFile(BytesIO(urllib.request.urlopen('http://md_file.zip').read()))
 df1 = pd.read_csv(zf.open('nm.dat'),header=None,delimiter='|', index_col=0, names=['aaa', 'xxxx', 'yyy','zzz'])
 df1.to_sql('nm',pg_engine,if_exists='replace')

python django python-3.x django-models

share|improve this question

edited Mar 26 at 0:29

Hybrid

3,94022 gold badges1010 silver badges3232 bronze badges

asked Mar 25 at 22:32

Bharat GudaBharat Guda

4977 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  why dont you save files you are fetching on local drive?
  
  – Yugandhar Chaudhari
  Mar 26 at 6:28
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  We wanted to automate that process and the company doesnt like anything to save on local
  
  – Bharat Guda
  Mar 26 at 16:05
  

  
  
  
  

add a comment | 

0

I am loading the website data into Postgres using Python-Django, there are around 9 csv files that i am loading to database but its taking so long ~10 hrs to fetch the data from website, and load it to Postgres. I wanted to increase the performance of the query. Can you guys please help with that?

This is just 1 dataframe, but i have 9 more similar to that, overall data is less than 500k records

from django.db import models 
from django_pandas.managers import DataFrameManager 
import pandas as pd 
from sqlalchemy import create_engine 
import zipfile 
import os 
from urllib.request import urlopen 
import urllib.request 
import io 
from io import BytesIO 

class mf(models.Model):
 pg_engine = create_engine('postgresql://user:password@server:host/db')
 zf = zipfile.ZipFile(BytesIO(urllib.request.urlopen('http://md_file.zip').read()))
 df1 = pd.read_csv(zf.open('nm.dat'),header=None,delimiter='|', index_col=0, names=['aaa', 'xxxx', 'yyy','zzz'])
 df1.to_sql('nm',pg_engine,if_exists='replace')

python django python-3.x django-models

share|improve this question

edited Mar 26 at 0:29

Hybrid

3,94022 gold badges1010 silver badges3232 bronze badges

asked Mar 25 at 22:32

Bharat GudaBharat Guda

4977 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  why dont you save files you are fetching on local drive?
  
  – Yugandhar Chaudhari
  Mar 26 at 6:28
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  We wanted to automate that process and the company doesnt like anything to save on local
  
  – Bharat Guda
  Mar 26 at 16:05
  

  
  
  
  

add a comment | 

I am loading the website data into Postgres using Python-Django, there are around 9 csv files that i am loading to database but its taking so long ~10 hrs to fetch the data from website, and load it to Postgres. I wanted to increase the performance of the query. Can you guys please help with that?

This is just 1 dataframe, but i have 9 more similar to that, overall data is less than 500k records

from django.db import models 
from django_pandas.managers import DataFrameManager 
import pandas as pd 
from sqlalchemy import create_engine 
import zipfile 
import os 
from urllib.request import urlopen 
import urllib.request 
import io 
from io import BytesIO 

class mf(models.Model):
 pg_engine = create_engine('postgresql://user:password@server:host/db')
 zf = zipfile.ZipFile(BytesIO(urllib.request.urlopen('http://md_file.zip').read()))
 df1 = pd.read_csv(zf.open('nm.dat'),header=None,delimiter='|', index_col=0, names=['aaa', 'xxxx', 'yyy','zzz'])
 df1.to_sql('nm',pg_engine,if_exists='replace')

python django python-3.x django-models

share|improve this question

edited Mar 26 at 0:29

Hybrid

3,94022 gold badges1010 silver badges3232 bronze badges

asked Mar 25 at 22:32

Bharat GudaBharat Guda

4977 bronze badges

from django.db import models 
from django_pandas.managers import DataFrameManager 
import pandas as pd 
from sqlalchemy import create_engine 
import zipfile 
import os 
from urllib.request import urlopen 
import urllib.request 
import io 
from io import BytesIO 

class mf(models.Model):
 pg_engine = create_engine('postgresql://user:password@server:host/db')
 zf = zipfile.ZipFile(BytesIO(urllib.request.urlopen('http://md_file.zip').read()))
 df1 = pd.read_csv(zf.open('nm.dat'),header=None,delimiter='|', index_col=0, names=['aaa', 'xxxx', 'yyy','zzz'])
 df1.to_sql('nm',pg_engine,if_exists='replace')

share|improve this question

edited Mar 26 at 0:29

Hybrid

3,94022 gold badges1010 silver badges3232 bronze badges

asked Mar 25 at 22:32

Bharat GudaBharat Guda

4977 bronze badges

share|improve this question

edited Mar 26 at 0:29

Hybrid

3,94022 gold badges1010 silver badges3232 bronze badges

asked Mar 25 at 22:32

Bharat GudaBharat Guda

4977 bronze badges

edited Mar 26 at 0:29

Hybrid

3,94022 gold badges1010 silver badges3232 bronze badges

edited Mar 26 at 0:29

Hybrid

3,94022 gold badges1010 silver badges3232 bronze badges

asked Mar 25 at 22:32

Bharat GudaBharat Guda

4977 bronze badges

asked Mar 25 at 22:32

Bharat GudaBharat Guda

4977 bronze badges