Memory error while doing Hierarchical ClusteringDistributed hierarchical clusteringHierarchical clustering of 1 million objectsCluster analysis in R: determine the optimal number of clustersMixed clustering (Kmeans + Hierarchical) in Weka?Hierarchical Cluster Analysis in Cluster 3.0Finding elements inside every cluster in scikit DBSCAN?Agglomerative hierarchical clustering techniquechoose cluster in hierarchical clusteringHierarchical Clustering with branching factor > 2?Hierarchical Clustering
Why are there five extra turns in tournament Magic?
What would a Dragon have to exhale to cause rain?
Do high-wing aircraft represent more difficult engineering challenges than low-wing aircraft?
Iterate lines of string variable in bash
Is there an academic word that means "to split hairs over"?
Why does the U.S military use mercenaries?
How was the blinking terminal cursor invented?
Why is vowel phonology represented in a trapezoid instead of a square?
Physically unpleasant work environment
How does the Heat Metal spell interact with a follow-up Frostbite spell?
Why were the bells ignored in S8E5?
Cannot remove door knob -- totally inaccessible!
SHAKE-128/256 or SHA3-256/512
Why is the marginal distribution/marginal probability described as "marginal"?
Is it possible to pass a pointer to an operator as an argument like a pointer to a function?
Why would company (decision makers) wait for someone to retire, rather than lay them off, when their role is no longer needed?
Write electromagnetic field tensor in terms of four-vector potential
Can I pay my credit card?
What color to choose as "danger" if the main color of my app is red
What is the purpose of : in math mode?
Is there any deeper thematic meaning to the white horse that Arya finds in The Bells (S08E05)?
Quadratic/polynomial problem
How come Arya Stark didn't burn in Game of Thrones Season 8 Episode 5
Do we see some Unsullied doing this in S08E05?
Memory error while doing Hierarchical Clustering
Distributed hierarchical clusteringHierarchical clustering of 1 million objectsCluster analysis in R: determine the optimal number of clustersMixed clustering (Kmeans + Hierarchical) in Weka?Hierarchical Cluster Analysis in Cluster 3.0Finding elements inside every cluster in scikit DBSCAN?Agglomerative hierarchical clustering techniquechoose cluster in hierarchical clusteringHierarchical Clustering with branching factor > 2?Hierarchical Clustering
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I have a large dataset (207989, 23), and I am trying to apply Hierarchical clustering on just one column right now to test if it's suitable for the task at my hand.
What I have tried:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import preprocessing
data = pd.read_csv('gpmd.csv', header = 0)
X = data.loc[:, ['ContextID', 'BacksGas_Flow_sccm']]
min_max_scaler = preprocessing.MinMaxScaler()
X_minmax = min_max_scaler.fit_transform(X.values[:,[1]])
import scipy.cluster.hierarchy as sch
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
after doing this, I am getting the following error:
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
Traceback (most recent call last):
File "<ipython-input-4-429f42b68112>", line 1, in <module>
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyclusterhierarchy.py", line 708, in linkage
y = distance.pdist(y, metric)
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyspatialdistance.py", line 1877, in pdist
dm = np.empty((m * (m - 1)) // 2, dtype=np.double)
MemoryError
Can someone explain what exactly is the problem here?
Thanks in advance
machine-learning cluster-analysis hierarchical-clustering
asked Mar 23 at 16:49
JunkratJunkrat
46013
add a comment |
I have a large dataset (207989, 23), and I am trying to apply Hierarchical clustering on just one column right now to test if it's suitable for the task at my hand.
What I have tried:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import preprocessing
data = pd.read_csv('gpmd.csv', header = 0)
X = data.loc[:, ['ContextID', 'BacksGas_Flow_sccm']]
min_max_scaler = preprocessing.MinMaxScaler()
X_minmax = min_max_scaler.fit_transform(X.values[:,[1]])
import scipy.cluster.hierarchy as sch
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
after doing this, I am getting the following error:
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
Traceback (most recent call last):
File "<ipython-input-4-429f42b68112>", line 1, in <module>
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyclusterhierarchy.py", line 708, in linkage
y = distance.pdist(y, metric)
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyspatialdistance.py", line 1877, in pdist
dm = np.empty((m * (m - 1)) // 2, dtype=np.double)
MemoryError
Can someone explain what exactly is the problem here?
Thanks in advance
machine-learning cluster-analysis hierarchical-clustering
asked Mar 23 at 16:49
JunkratJunkrat
46013
1
The problem is thatsch.linkagehas quadratic memory complexity in terms of the number of original observations. As you have quite a number of them (207989) you should consider trying less memory demanding algorithms.
– Mikhail Berlinkov
Mar 23 at 16:59
Hey, @MikhailBerlinkov .In other words, you mean to say that Hierarchical clustering isn't suitable for large datasets?
– Junkrat
Mar 23 at 17:04
No, I meant only this particular algorithm is quadratic.
– Mikhail Berlinkov
Mar 24 at 18:12
add a comment |
I have a large dataset (207989, 23), and I am trying to apply Hierarchical clustering on just one column right now to test if it's suitable for the task at my hand.
What I have tried:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import preprocessing
data = pd.read_csv('gpmd.csv', header = 0)
X = data.loc[:, ['ContextID', 'BacksGas_Flow_sccm']]
min_max_scaler = preprocessing.MinMaxScaler()
X_minmax = min_max_scaler.fit_transform(X.values[:,[1]])
import scipy.cluster.hierarchy as sch
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
after doing this, I am getting the following error:
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
Traceback (most recent call last):
File "<ipython-input-4-429f42b68112>", line 1, in <module>
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyclusterhierarchy.py", line 708, in linkage
y = distance.pdist(y, metric)
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyspatialdistance.py", line 1877, in pdist
dm = np.empty((m * (m - 1)) // 2, dtype=np.double)
MemoryError
Can someone explain what exactly is the problem here?
Thanks in advance
machine-learning cluster-analysis hierarchical-clustering
asked Mar 23 at 16:49
JunkratJunkrat
46013
I have a large dataset (207989, 23), and I am trying to apply Hierarchical clustering on just one column right now to test if it's suitable for the task at my hand.
What I have tried:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import preprocessing
data = pd.read_csv('gpmd.csv', header = 0)
X = data.loc[:, ['ContextID', 'BacksGas_Flow_sccm']]
min_max_scaler = preprocessing.MinMaxScaler()
X_minmax = min_max_scaler.fit_transform(X.values[:,[1]])
import scipy.cluster.hierarchy as sch
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
after doing this, I am getting the following error:
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
Traceback (most recent call last):
File "<ipython-input-4-429f42b68112>", line 1, in <module>
dendrogram = sch.dendrogram(sch.linkage(X_minmax, method = 'ward'))
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyclusterhierarchy.py", line 708, in linkage
y = distance.pdist(y, metric)
File "C:UserskashyAnaconda3envspy36libsite-packagesscipyspatialdistance.py", line 1877, in pdist
dm = np.empty((m * (m - 1)) // 2, dtype=np.double)
MemoryError
Can someone explain what exactly is the problem here?
Thanks in advance
machine-learning cluster-analysis hierarchical-clustering
machine-learning cluster-analysis hierarchical-clustering
asked Mar 23 at 16:49
JunkratJunkrat
46013
asked Mar 23 at 16:49
JunkratJunkrat
46013
asked Mar 23 at 16:49
JunkratJunkrat
46013
asked Mar 23 at 16:49
JunkratJunkrat
46013
asked Mar 23 at 16:49
JunkratJunkrat
46013
46013
1
The problem is thatsch.linkagehas quadratic memory complexity in terms of the number of original observations. As you have quite a number of them (207989) you should consider trying less memory demanding algorithms.
– Mikhail Berlinkov
Mar 23 at 16:59
Hey, @MikhailBerlinkov .In other words, you mean to say that Hierarchical clustering isn't suitable for large datasets?
– Junkrat
Mar 23 at 17:04
No, I meant only this particular algorithm is quadratic.
– Mikhail Berlinkov
Mar 24 at 18:12
add a comment |
1
The problem is thatsch.linkagehas quadratic memory complexity in terms of the number of original observations. As you have quite a number of them (207989) you should consider trying less memory demanding algorithms.
– Mikhail Berlinkov
Mar 23 at 16:59
Hey, @MikhailBerlinkov .In other words, you mean to say that Hierarchical clustering isn't suitable for large datasets?
– Junkrat
Mar 23 at 17:04
No, I meant only this particular algorithm is quadratic.
– Mikhail Berlinkov
Mar 24 at 18:12
1
1
The problem is that
sch.linkage has quadratic memory complexity in terms of the number of original observations. As you have quite a number of them (207989) you should consider trying less memory demanding algorithms.– Mikhail Berlinkov
Mar 23 at 16:59
The problem is that
sch.linkage has quadratic memory complexity in terms of the number of original observations. As you have quite a number of them (207989) you should consider trying less memory demanding algorithms.– Mikhail Berlinkov
Mar 23 at 16:59
Hey, @MikhailBerlinkov .In other words, you mean to say that Hierarchical clustering isn't suitable for large datasets?
– Junkrat
Mar 23 at 17:04
Hey, @MikhailBerlinkov .In other words, you mean to say that Hierarchical clustering isn't suitable for large datasets?
– Junkrat
Mar 23 at 17:04
No, I meant only this particular algorithm is quadratic.
– Mikhail Berlinkov
Mar 24 at 18:12
No, I meant only this particular algorithm is quadratic.
– Mikhail Berlinkov
Mar 24 at 18:12
add a comment |
1 Answer
1
active
oldest
votes
Hierarchical clustering in most variants needs O(n²) memory.
Because of this, most implementations will fail at around 65535 instances, when they hit the 32 bit mark (some may fail at 32k already). But just do the math: n * n * 8 bytes for double precision: how much memory would you need?
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55316093%2fmemory-error-while-doing-hierarchical-clustering%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Hierarchical clustering in most variants needs O(n²) memory.
Because of this, most implementations will fail at around 65535 instances, when they hit the 32 bit mark (some may fail at 32k already). But just do the math: n * n * 8 bytes for double precision: how much memory would you need?
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
add a comment |
Hierarchical clustering in most variants needs O(n²) memory.
Because of this, most implementations will fail at around 65535 instances, when they hit the 32 bit mark (some may fail at 32k already). But just do the math: n * n * 8 bytes for double precision: how much memory would you need?
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
add a comment |
Hierarchical clustering in most variants needs O(n²) memory.
Because of this, most implementations will fail at around 65535 instances, when they hit the 32 bit mark (some may fail at 32k already). But just do the math: n * n * 8 bytes for double precision: how much memory would you need?
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
Hierarchical clustering in most variants needs O(n²) memory.
Because of this, most implementations will fail at around 65535 instances, when they hit the 32 bit mark (some may fail at 32k already). But just do the math: n * n * 8 bytes for double precision: how much memory would you need?
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
answered Mar 23 at 23:17
Anony-MousseAnony-Mousse
59.9k799164
59.9k799164
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55316093%2fmemory-error-while-doing-hierarchical-clustering%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
The problem is that
sch.linkagehas quadratic memory complexity in terms of the number of original observations. As you have quite a number of them (207989) you should consider trying less memory demanding algorithms.– Mikhail Berlinkov
Mar 23 at 16:59
Hey, @MikhailBerlinkov .In other words, you mean to say that Hierarchical clustering isn't suitable for large datasets?
– Junkrat
Mar 23 at 17:04
No, I meant only this particular algorithm is quadratic.
– Mikhail Berlinkov
Mar 24 at 18:12