Is MongoDB faster for storing data than pandas read/write to_excel?NoSQL (MongoDB) vs Lucene (or Solr) as your databaseIs a file read faster than reading data from the database?“Large data” work flows using pandasChange data type of columns in PandasWriting a pandas DataFrame to CSV fileUsing Pandas to pd.read_excel() for multiple worksheets of the same workbookwrite data column wise in a MongoDB documentInsert data at a particular excel cell using pandasHow do I use python pandas to read an already opened excel sheetRead and Write to MySQL Database with Python Pandas
Is it ok for parents to kiss and romance with each other while their 2- to 8-year-old child watches?
How do I explain that I don't want to maintain old projects?
US citizen traveling with Peruvian passport
Did Rabbi Akiva accept arguments from ignorance?
What could cause the sea level to massively decrease?
Conditions for Roots of a quadratic equation at infinity
Found and corrected a mistake on someone's else paper -- praxis?
What are the consequences for a developed nation to not accept any refugees?
Why does the Antonov AN-225 not have any winglets?
No Torah = Revert to Nothingness?
Is it possible for a character at any level to cast all 44 Cantrips in one week without Magic Items?
Estimates on number of topologies on a finite set
Optimization models for portfolio optimization
Can a landlord force all residents to use the landlord's in-house debit card accounts?
When do flights get cancelled due to fog?
This LM317 diagram doesn't make any sense to me
Distinguish the explanations of Galadriel's test in LotR
Red token deck mass token destruction enchantment protection mtg
Generalized Behrend version for Grothendieck-Lefschetz trace formula
What is the meaning of “Can I have a slice?” In NYC?
Is this Cambridge Dictionary example of "felicitate" valid?
How insert vertex in face?
Swapping "Good" and "Bad"
Hail hit my roof. Do I need to replace it?
Is MongoDB faster for storing data than pandas read/write to_excel?
NoSQL (MongoDB) vs Lucene (or Solr) as your databaseIs a file read faster than reading data from the database?“Large data” work flows using pandasChange data type of columns in PandasWriting a pandas DataFrame to CSV fileUsing Pandas to pd.read_excel() for multiple worksheets of the same workbookwrite data column wise in a MongoDB documentInsert data at a particular excel cell using pandasHow do I use python pandas to read an already opened excel sheetRead and Write to MySQL Database with Python Pandas
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I am developing a database of information and currently storing the data in various sheet in multiple XLSX files and considering switching my data to a MongoDB database. Is it quicker read/write for MongoDB than read/write for Pandas? Is Pandas just a nice way to analyze a relational database versus storing information in excel files?
EDIT:
Suppose I have an excel file with multiple sheets and each sheet has 1,000+ lines of floating point data with 5 columns and I read in the XLSX, parse it.
df = pd.read_excel("myfile.xlsx")
would it be faster/more efficient to pull the data directly from a mongoDB database than the read/write of Pandas.
python-3.x mongodb pandas
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
add a comment |
I am developing a database of information and currently storing the data in various sheet in multiple XLSX files and considering switching my data to a MongoDB database. Is it quicker read/write for MongoDB than read/write for Pandas? Is Pandas just a nice way to analyze a relational database versus storing information in excel files?
EDIT:
Suppose I have an excel file with multiple sheets and each sheet has 1,000+ lines of floating point data with 5 columns and I read in the XLSX, parse it.
df = pd.read_excel("myfile.xlsx")
would it be faster/more efficient to pull the data directly from a mongoDB database than the read/write of Pandas.
python-3.x mongodb pandas
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
1
I'm not sure what the ask is. "Quicker" is always relative to something, e.g. your hardware, your use case, etc. "Database of information" is not very specific. Pandas is mainly a data analysis tool, it's unclear why you want to use Pandas to store anything. Please rephrase the question in an actionable manner, e.g. where the answer is not opinion-based.
– kevinadi
Mar 26 at 2:10
I am specifically speaking of storing historical price data from Bittrex in MongoDB versus storing it in an excel file
– Bryce Wayne
Mar 26 at 4:24
1
From your updated question: all things being equal (hardware etc.), getting data from a database will almost always be faster than reading an excel file (assuming the proper schema design, among others). This is what a database is designed for. Reading excel file will always involve reading and interpreting the contents, which is at least another layer of processing. However this is assuming many things are equal, which may or may not be the case in your environment.
– kevinadi
Mar 26 at 5:08
add a comment |
I am developing a database of information and currently storing the data in various sheet in multiple XLSX files and considering switching my data to a MongoDB database. Is it quicker read/write for MongoDB than read/write for Pandas? Is Pandas just a nice way to analyze a relational database versus storing information in excel files?
EDIT:
Suppose I have an excel file with multiple sheets and each sheet has 1,000+ lines of floating point data with 5 columns and I read in the XLSX, parse it.
df = pd.read_excel("myfile.xlsx")
would it be faster/more efficient to pull the data directly from a mongoDB database than the read/write of Pandas.
python-3.x mongodb pandas
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
I am developing a database of information and currently storing the data in various sheet in multiple XLSX files and considering switching my data to a MongoDB database. Is it quicker read/write for MongoDB than read/write for Pandas? Is Pandas just a nice way to analyze a relational database versus storing information in excel files?
EDIT:
Suppose I have an excel file with multiple sheets and each sheet has 1,000+ lines of floating point data with 5 columns and I read in the XLSX, parse it.
df = pd.read_excel("myfile.xlsx")
would it be faster/more efficient to pull the data directly from a mongoDB database than the read/write of Pandas.
python-3.x mongodb pandas
python-3.x mongodb pandas
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
edited Mar 26 at 4:29
Bryce Wayne
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
asked Mar 25 at 23:23
Bryce WayneBryce Wayne
1471 silver badge10 bronze badges
1471 silver badge10 bronze badges
1
I'm not sure what the ask is. "Quicker" is always relative to something, e.g. your hardware, your use case, etc. "Database of information" is not very specific. Pandas is mainly a data analysis tool, it's unclear why you want to use Pandas to store anything. Please rephrase the question in an actionable manner, e.g. where the answer is not opinion-based.
– kevinadi
Mar 26 at 2:10
I am specifically speaking of storing historical price data from Bittrex in MongoDB versus storing it in an excel file
– Bryce Wayne
Mar 26 at 4:24
1
From your updated question: all things being equal (hardware etc.), getting data from a database will almost always be faster than reading an excel file (assuming the proper schema design, among others). This is what a database is designed for. Reading excel file will always involve reading and interpreting the contents, which is at least another layer of processing. However this is assuming many things are equal, which may or may not be the case in your environment.
– kevinadi
Mar 26 at 5:08
add a comment |
1
I'm not sure what the ask is. "Quicker" is always relative to something, e.g. your hardware, your use case, etc. "Database of information" is not very specific. Pandas is mainly a data analysis tool, it's unclear why you want to use Pandas to store anything. Please rephrase the question in an actionable manner, e.g. where the answer is not opinion-based.
– kevinadi
Mar 26 at 2:10
I am specifically speaking of storing historical price data from Bittrex in MongoDB versus storing it in an excel file
– Bryce Wayne
Mar 26 at 4:24
1
From your updated question: all things being equal (hardware etc.), getting data from a database will almost always be faster than reading an excel file (assuming the proper schema design, among others). This is what a database is designed for. Reading excel file will always involve reading and interpreting the contents, which is at least another layer of processing. However this is assuming many things are equal, which may or may not be the case in your environment.
– kevinadi
Mar 26 at 5:08
1
1
I'm not sure what the ask is. "Quicker" is always relative to something, e.g. your hardware, your use case, etc. "Database of information" is not very specific. Pandas is mainly a data analysis tool, it's unclear why you want to use Pandas to store anything. Please rephrase the question in an actionable manner, e.g. where the answer is not opinion-based.
– kevinadi
Mar 26 at 2:10
I'm not sure what the ask is. "Quicker" is always relative to something, e.g. your hardware, your use case, etc. "Database of information" is not very specific. Pandas is mainly a data analysis tool, it's unclear why you want to use Pandas to store anything. Please rephrase the question in an actionable manner, e.g. where the answer is not opinion-based.
– kevinadi
Mar 26 at 2:10
I am specifically speaking of storing historical price data from Bittrex in MongoDB versus storing it in an excel file
– Bryce Wayne
Mar 26 at 4:24
I am specifically speaking of storing historical price data from Bittrex in MongoDB versus storing it in an excel file
– Bryce Wayne
Mar 26 at 4:24
1
1
From your updated question: all things being equal (hardware etc.), getting data from a database will almost always be faster than reading an excel file (assuming the proper schema design, among others). This is what a database is designed for. Reading excel file will always involve reading and interpreting the contents, which is at least another layer of processing. However this is assuming many things are equal, which may or may not be the case in your environment.
– kevinadi
Mar 26 at 5:08
From your updated question: all things being equal (hardware etc.), getting data from a database will almost always be faster than reading an excel file (assuming the proper schema design, among others). This is what a database is designed for. Reading excel file will always involve reading and interpreting the contents, which is at least another layer of processing. However this is assuming many things are equal, which may or may not be the case in your environment.
– kevinadi
Mar 26 at 5:08
add a comment |
1 Answer
1
active
oldest
votes
Use timeit to compare running time of the code snippet you provided,
on your hardware,
with running time of a similar mongo snippet,
on your hardware.
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55347792%2fis-mongodb-faster-for-storing-data-than-pandas-read-write-to-excel%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Use timeit to compare running time of the code snippet you provided,
on your hardware,
with running time of a similar mongo snippet,
on your hardware.
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
add a comment |
Use timeit to compare running time of the code snippet you provided,
on your hardware,
with running time of a similar mongo snippet,
on your hardware.
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
add a comment |
Use timeit to compare running time of the code snippet you provided,
on your hardware,
with running time of a similar mongo snippet,
on your hardware.
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
Use timeit to compare running time of the code snippet you provided,
on your hardware,
with running time of a similar mongo snippet,
on your hardware.
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
answered Mar 26 at 5:58
J_HJ_H
5,8161 gold badge9 silver badges23 bronze badges
5,8161 gold badge9 silver badges23 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55347792%2fis-mongodb-faster-for-storing-data-than-pandas-read-write-to-excel%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
I'm not sure what the ask is. "Quicker" is always relative to something, e.g. your hardware, your use case, etc. "Database of information" is not very specific. Pandas is mainly a data analysis tool, it's unclear why you want to use Pandas to store anything. Please rephrase the question in an actionable manner, e.g. where the answer is not opinion-based.
– kevinadi
Mar 26 at 2:10
I am specifically speaking of storing historical price data from Bittrex in MongoDB versus storing it in an excel file
– Bryce Wayne
Mar 26 at 4:24
1
From your updated question: all things being equal (hardware etc.), getting data from a database will almost always be faster than reading an excel file (assuming the proper schema design, among others). This is what a database is designed for. Reading excel file will always involve reading and interpreting the contents, which is at least another layer of processing. However this is assuming many things are equal, which may or may not be the case in your environment.
– kevinadi
Mar 26 at 5:08