Role of std::unordered_set::reserve for container memory requirement?How to measure Memory Usage of std::unordered_mapstd::deque memory usage - Visual C++, and comparison to othersstd::string implementation in GCC and its memory overhead for short stringslambda functors assignment workaroundDoes the standard guarantee that the total memory occupied by a std::vector scales as C+N*sizeof(T)?Minimizing memory overhead using C++ containers (std::map and std::vector too expensive)How to extend std library containers to get notified when individual elements are changedwhy does std::allocator::deallocate require a size?Memory-efficient custom deleter for std::unique_ptr?Will C++ STL vector reserving too many capacity costs lots of memory?Simplest way to get memory size of std::array's underlying array?
(11 of 11: Meta) What is Pyramid Cult's All-Time Favorite?
Is it okay for a ticket seller to grab a tip in the USA?
Best gun to modify into a monsterhunter weapon?
Withdrew when Jimmy met up with Heath
How are you supposed to know the strumming pattern for a song from the "chord sheet music"?
What skills in 5e give trap knowledge (i.e. the equivalent of Dungeoneering in 4e)?
Trying to write a shell script that keeps testing a server remotely, but it keeps falling in else statement when I logout
Should RL rewards diminish over time?
What is my malfunctioning AI harvesting from humans?
What does "目的化して久しい" mean in this dialogue?
What are the uses and limitations of Persuasion, Insight, and Deception against other PCs?
how to differentiate when a child lwc component is called twice in parent component?
Why isn’t SHA-3 in wider use?
Identification of vintage sloping window
The cat ate your input again!
Are differences between uniformly distributed numbers uniformly distributed?
What does this double-treble double-bass staff mean?
How quickly could a country build a tall concrete wall around a city?
Simple Stop watch which i want to extend
How can you evade tax by getting employment income just in equity, then using this equity as collateral to take out loan?
I accidentally overwrote a Linux binary file
What does "sardine box" mean?
In a topological space if there exists a loop that cannot be contracted to a point does there exist a simple loop that cannot be contracted also?
How to create all combinations from a nested list while preserving the structure using R?
Role of std::unordered_set::reserve for container memory requirement?
How to measure Memory Usage of std::unordered_mapstd::deque memory usage - Visual C++, and comparison to othersstd::string implementation in GCC and its memory overhead for short stringslambda functors assignment workaroundDoes the standard guarantee that the total memory occupied by a std::vector scales as C+N*sizeof(T)?Minimizing memory overhead using C++ containers (std::map and std::vector too expensive)How to extend std library containers to get notified when individual elements are changedwhy does std::allocator::deallocate require a size?Memory-efficient custom deleter for std::unique_ptr?Will C++ STL vector reserving too many capacity costs lots of memory?Simplest way to get memory size of std::array's underlying array?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
Suppose I use a std::unordered_set<MyClass>, and suppose sizeof(MyClass) is large, i.e. much larger than sizeof(size_t) and sizeof(void*). I will add a large number numberOfElementsToBeAdded of elements to the unordered set.
From https://stackoverflow.com/a/25438497/4237 , I find:
"each memory allocation may be rounded up to a size convenient for the memory allocation library's management - e.g. the next power of two, which approaches 100% worst case inefficiency and 50% average, so let's add 50%, just for the list nodes as the buckets are likely powers of two given size_t and a pointer: 50% * size() * (sizeof(void*) + sizeof((M::value_type))"
This drives me to conclude that actual memory consumption will be between 1*numberOfElements*sizeof(MyClass) and (1+1)*numberOfElements*sizeof(MyClass), modulo some additional memory, which is of little interest here because it is of the order sizeof(size_t).
However, I know the number of elements to be added in advance, so I call:
std::unordered_set<MyClass> set;
set.reserve(numberOfElementsToBeAdded);
//Insert elements
Thinking about the parallel of calling std::vector::reserve, I would guess that this avoids the potential overhead, and would thus guarantee memory consumption to be approximately 1*numberOfElements*sizeof(MyClass) (modulo some additional memory, again of order sizeof(size_t)).
Could I rely on implementations of the standard library to guarantee that calling reserve like this will avoid the 0-100% (50% average) overhead mentioned in the above answer?
c++ stl containers std
add a comment |
Suppose I use a std::unordered_set<MyClass>, and suppose sizeof(MyClass) is large, i.e. much larger than sizeof(size_t) and sizeof(void*). I will add a large number numberOfElementsToBeAdded of elements to the unordered set.
From https://stackoverflow.com/a/25438497/4237 , I find:
"each memory allocation may be rounded up to a size convenient for the memory allocation library's management - e.g. the next power of two, which approaches 100% worst case inefficiency and 50% average, so let's add 50%, just for the list nodes as the buckets are likely powers of two given size_t and a pointer: 50% * size() * (sizeof(void*) + sizeof((M::value_type))"
This drives me to conclude that actual memory consumption will be between 1*numberOfElements*sizeof(MyClass) and (1+1)*numberOfElements*sizeof(MyClass), modulo some additional memory, which is of little interest here because it is of the order sizeof(size_t).
However, I know the number of elements to be added in advance, so I call:
std::unordered_set<MyClass> set;
set.reserve(numberOfElementsToBeAdded);
//Insert elements
Thinking about the parallel of calling std::vector::reserve, I would guess that this avoids the potential overhead, and would thus guarantee memory consumption to be approximately 1*numberOfElements*sizeof(MyClass) (modulo some additional memory, again of order sizeof(size_t)).
Could I rely on implementations of the standard library to guarantee that calling reserve like this will avoid the 0-100% (50% average) overhead mentioned in the above answer?
c++ stl containers std
add a comment |
Suppose I use a std::unordered_set<MyClass>, and suppose sizeof(MyClass) is large, i.e. much larger than sizeof(size_t) and sizeof(void*). I will add a large number numberOfElementsToBeAdded of elements to the unordered set.
From https://stackoverflow.com/a/25438497/4237 , I find:
"each memory allocation may be rounded up to a size convenient for the memory allocation library's management - e.g. the next power of two, which approaches 100% worst case inefficiency and 50% average, so let's add 50%, just for the list nodes as the buckets are likely powers of two given size_t and a pointer: 50% * size() * (sizeof(void*) + sizeof((M::value_type))"
This drives me to conclude that actual memory consumption will be between 1*numberOfElements*sizeof(MyClass) and (1+1)*numberOfElements*sizeof(MyClass), modulo some additional memory, which is of little interest here because it is of the order sizeof(size_t).
However, I know the number of elements to be added in advance, so I call:
std::unordered_set<MyClass> set;
set.reserve(numberOfElementsToBeAdded);
//Insert elements
Thinking about the parallel of calling std::vector::reserve, I would guess that this avoids the potential overhead, and would thus guarantee memory consumption to be approximately 1*numberOfElements*sizeof(MyClass) (modulo some additional memory, again of order sizeof(size_t)).
Could I rely on implementations of the standard library to guarantee that calling reserve like this will avoid the 0-100% (50% average) overhead mentioned in the above answer?
c++ stl containers std
Suppose I use a std::unordered_set<MyClass>, and suppose sizeof(MyClass) is large, i.e. much larger than sizeof(size_t) and sizeof(void*). I will add a large number numberOfElementsToBeAdded of elements to the unordered set.
From https://stackoverflow.com/a/25438497/4237 , I find:
"each memory allocation may be rounded up to a size convenient for the memory allocation library's management - e.g. the next power of two, which approaches 100% worst case inefficiency and 50% average, so let's add 50%, just for the list nodes as the buckets are likely powers of two given size_t and a pointer: 50% * size() * (sizeof(void*) + sizeof((M::value_type))"
This drives me to conclude that actual memory consumption will be between 1*numberOfElements*sizeof(MyClass) and (1+1)*numberOfElements*sizeof(MyClass), modulo some additional memory, which is of little interest here because it is of the order sizeof(size_t).
However, I know the number of elements to be added in advance, so I call:
std::unordered_set<MyClass> set;
set.reserve(numberOfElementsToBeAdded);
//Insert elements
Thinking about the parallel of calling std::vector::reserve, I would guess that this avoids the potential overhead, and would thus guarantee memory consumption to be approximately 1*numberOfElements*sizeof(MyClass) (modulo some additional memory, again of order sizeof(size_t)).
Could I rely on implementations of the standard library to guarantee that calling reserve like this will avoid the 0-100% (50% average) overhead mentioned in the above answer?
c++ stl containers std
c++ stl containers std
edited Mar 27 at 14:39
willem
asked Mar 27 at 8:23
willemwillem
1,2793 gold badges17 silver badges36 bronze badges
1,2793 gold badges17 silver badges36 bronze badges
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
From this std::unordered_set::reserve reference
Sets the number of buckets to the number needed to accomodate at least
countelements ...
There's some more, but the gist is that std::unordered_set::reserve doesn't actually work like std::vector::reserve. For an unordered set it allocates buckets for the hash table, not memory for the actual elements themselves.
The set could put one element per bucket, but it's not guaranteed.
1
Right. The purpose of callingstd::unordered_set::reserveis to avoid having to re-size the hashing structures used internally for indexing the set. It (typically) has no effect on how the actual members of the set are allocated or stored.
– David Schwartz
Mar 27 at 8:35
Do buckets only require sizeof(size_t)/sizeof(void*) memory or do they (also) require sizeof(M:valuetype) = sizeof(MyClass) memory?
– willem
Mar 27 at 8:44
@willem That's an implementation-specific detail that is not specified. You have to check your implementation.
– Some programmer dude
Mar 27 at 8:55
@Someprogrammerdude Thanks. You say "the set could put one element per bucket, but it's not guaranteed." I have trouble understanding what that implies. Am I correct in saying that it could also be less than 1 element/MyClass per bucket?
– willem
Mar 27 at 12:50
1
@willem I don't know how much memory an empty bucket would use, since that depends on the implementation. It also feels like you're not telling us the whole problem... Why do you need to know? What is the actual problem you need to solve? Why are you so worried about memory consumption (especially since we're still in the realm of low single-digit MiB range here)?
– Some programmer dude
Mar 28 at 8:43
|
show 5 more comments
Quoting the standard:
Sets the number of buckets to the number needed to accomodate at least
count elements without exceeding maximum load factor and rehashes the
container, i.e. puts the elements into appropriate buckets considering
that total number of buckets has changed
The important part is **at least*. I do not think you can rely on the implementation and be sure that it only uses less amount of memory. My guess is that regardless if you call reserve the memory usage for numElements will be similar.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55372630%2frole-of-stdunordered-setreserve-for-container-memory-requirement%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
From this std::unordered_set::reserve reference
Sets the number of buckets to the number needed to accomodate at least
countelements ...
There's some more, but the gist is that std::unordered_set::reserve doesn't actually work like std::vector::reserve. For an unordered set it allocates buckets for the hash table, not memory for the actual elements themselves.
The set could put one element per bucket, but it's not guaranteed.
1
Right. The purpose of callingstd::unordered_set::reserveis to avoid having to re-size the hashing structures used internally for indexing the set. It (typically) has no effect on how the actual members of the set are allocated or stored.
– David Schwartz
Mar 27 at 8:35
Do buckets only require sizeof(size_t)/sizeof(void*) memory or do they (also) require sizeof(M:valuetype) = sizeof(MyClass) memory?
– willem
Mar 27 at 8:44
@willem That's an implementation-specific detail that is not specified. You have to check your implementation.
– Some programmer dude
Mar 27 at 8:55
@Someprogrammerdude Thanks. You say "the set could put one element per bucket, but it's not guaranteed." I have trouble understanding what that implies. Am I correct in saying that it could also be less than 1 element/MyClass per bucket?
– willem
Mar 27 at 12:50
1
@willem I don't know how much memory an empty bucket would use, since that depends on the implementation. It also feels like you're not telling us the whole problem... Why do you need to know? What is the actual problem you need to solve? Why are you so worried about memory consumption (especially since we're still in the realm of low single-digit MiB range here)?
– Some programmer dude
Mar 28 at 8:43
|
show 5 more comments
From this std::unordered_set::reserve reference
Sets the number of buckets to the number needed to accomodate at least
countelements ...
There's some more, but the gist is that std::unordered_set::reserve doesn't actually work like std::vector::reserve. For an unordered set it allocates buckets for the hash table, not memory for the actual elements themselves.
The set could put one element per bucket, but it's not guaranteed.
1
Right. The purpose of callingstd::unordered_set::reserveis to avoid having to re-size the hashing structures used internally for indexing the set. It (typically) has no effect on how the actual members of the set are allocated or stored.
– David Schwartz
Mar 27 at 8:35
Do buckets only require sizeof(size_t)/sizeof(void*) memory or do they (also) require sizeof(M:valuetype) = sizeof(MyClass) memory?
– willem
Mar 27 at 8:44
@willem That's an implementation-specific detail that is not specified. You have to check your implementation.
– Some programmer dude
Mar 27 at 8:55
@Someprogrammerdude Thanks. You say "the set could put one element per bucket, but it's not guaranteed." I have trouble understanding what that implies. Am I correct in saying that it could also be less than 1 element/MyClass per bucket?
– willem
Mar 27 at 12:50
1
@willem I don't know how much memory an empty bucket would use, since that depends on the implementation. It also feels like you're not telling us the whole problem... Why do you need to know? What is the actual problem you need to solve? Why are you so worried about memory consumption (especially since we're still in the realm of low single-digit MiB range here)?
– Some programmer dude
Mar 28 at 8:43
|
show 5 more comments
From this std::unordered_set::reserve reference
Sets the number of buckets to the number needed to accomodate at least
countelements ...
There's some more, but the gist is that std::unordered_set::reserve doesn't actually work like std::vector::reserve. For an unordered set it allocates buckets for the hash table, not memory for the actual elements themselves.
The set could put one element per bucket, but it's not guaranteed.
From this std::unordered_set::reserve reference
Sets the number of buckets to the number needed to accomodate at least
countelements ...
There's some more, but the gist is that std::unordered_set::reserve doesn't actually work like std::vector::reserve. For an unordered set it allocates buckets for the hash table, not memory for the actual elements themselves.
The set could put one element per bucket, but it's not guaranteed.
edited Mar 27 at 8:36
answered Mar 27 at 8:34
Some programmer dudeSome programmer dude
312k26 gold badges281 silver badges449 bronze badges
312k26 gold badges281 silver badges449 bronze badges
1
Right. The purpose of callingstd::unordered_set::reserveis to avoid having to re-size the hashing structures used internally for indexing the set. It (typically) has no effect on how the actual members of the set are allocated or stored.
– David Schwartz
Mar 27 at 8:35
Do buckets only require sizeof(size_t)/sizeof(void*) memory or do they (also) require sizeof(M:valuetype) = sizeof(MyClass) memory?
– willem
Mar 27 at 8:44
@willem That's an implementation-specific detail that is not specified. You have to check your implementation.
– Some programmer dude
Mar 27 at 8:55
@Someprogrammerdude Thanks. You say "the set could put one element per bucket, but it's not guaranteed." I have trouble understanding what that implies. Am I correct in saying that it could also be less than 1 element/MyClass per bucket?
– willem
Mar 27 at 12:50
1
@willem I don't know how much memory an empty bucket would use, since that depends on the implementation. It also feels like you're not telling us the whole problem... Why do you need to know? What is the actual problem you need to solve? Why are you so worried about memory consumption (especially since we're still in the realm of low single-digit MiB range here)?
– Some programmer dude
Mar 28 at 8:43
|
show 5 more comments
1
Right. The purpose of callingstd::unordered_set::reserveis to avoid having to re-size the hashing structures used internally for indexing the set. It (typically) has no effect on how the actual members of the set are allocated or stored.
– David Schwartz
Mar 27 at 8:35
Do buckets only require sizeof(size_t)/sizeof(void*) memory or do they (also) require sizeof(M:valuetype) = sizeof(MyClass) memory?
– willem
Mar 27 at 8:44
@willem That's an implementation-specific detail that is not specified. You have to check your implementation.
– Some programmer dude
Mar 27 at 8:55
@Someprogrammerdude Thanks. You say "the set could put one element per bucket, but it's not guaranteed." I have trouble understanding what that implies. Am I correct in saying that it could also be less than 1 element/MyClass per bucket?
– willem
Mar 27 at 12:50
1
@willem I don't know how much memory an empty bucket would use, since that depends on the implementation. It also feels like you're not telling us the whole problem... Why do you need to know? What is the actual problem you need to solve? Why are you so worried about memory consumption (especially since we're still in the realm of low single-digit MiB range here)?
– Some programmer dude
Mar 28 at 8:43
1
1
Right. The purpose of calling
std::unordered_set::reserve is to avoid having to re-size the hashing structures used internally for indexing the set. It (typically) has no effect on how the actual members of the set are allocated or stored.– David Schwartz
Mar 27 at 8:35
Right. The purpose of calling
std::unordered_set::reserve is to avoid having to re-size the hashing structures used internally for indexing the set. It (typically) has no effect on how the actual members of the set are allocated or stored.– David Schwartz
Mar 27 at 8:35
Do buckets only require sizeof(size_t)/sizeof(void*) memory or do they (also) require sizeof(M:valuetype) = sizeof(MyClass) memory?
– willem
Mar 27 at 8:44
Do buckets only require sizeof(size_t)/sizeof(void*) memory or do they (also) require sizeof(M:valuetype) = sizeof(MyClass) memory?
– willem
Mar 27 at 8:44
@willem That's an implementation-specific detail that is not specified. You have to check your implementation.
– Some programmer dude
Mar 27 at 8:55
@willem That's an implementation-specific detail that is not specified. You have to check your implementation.
– Some programmer dude
Mar 27 at 8:55
@Someprogrammerdude Thanks. You say "the set could put one element per bucket, but it's not guaranteed." I have trouble understanding what that implies. Am I correct in saying that it could also be less than 1 element/MyClass per bucket?
– willem
Mar 27 at 12:50
@Someprogrammerdude Thanks. You say "the set could put one element per bucket, but it's not guaranteed." I have trouble understanding what that implies. Am I correct in saying that it could also be less than 1 element/MyClass per bucket?
– willem
Mar 27 at 12:50
1
1
@willem I don't know how much memory an empty bucket would use, since that depends on the implementation. It also feels like you're not telling us the whole problem... Why do you need to know? What is the actual problem you need to solve? Why are you so worried about memory consumption (especially since we're still in the realm of low single-digit MiB range here)?
– Some programmer dude
Mar 28 at 8:43
@willem I don't know how much memory an empty bucket would use, since that depends on the implementation. It also feels like you're not telling us the whole problem... Why do you need to know? What is the actual problem you need to solve? Why are you so worried about memory consumption (especially since we're still in the realm of low single-digit MiB range here)?
– Some programmer dude
Mar 28 at 8:43
|
show 5 more comments
Quoting the standard:
Sets the number of buckets to the number needed to accomodate at least
count elements without exceeding maximum load factor and rehashes the
container, i.e. puts the elements into appropriate buckets considering
that total number of buckets has changed
The important part is **at least*. I do not think you can rely on the implementation and be sure that it only uses less amount of memory. My guess is that regardless if you call reserve the memory usage for numElements will be similar.
add a comment |
Quoting the standard:
Sets the number of buckets to the number needed to accomodate at least
count elements without exceeding maximum load factor and rehashes the
container, i.e. puts the elements into appropriate buckets considering
that total number of buckets has changed
The important part is **at least*. I do not think you can rely on the implementation and be sure that it only uses less amount of memory. My guess is that regardless if you call reserve the memory usage for numElements will be similar.
add a comment |
Quoting the standard:
Sets the number of buckets to the number needed to accomodate at least
count elements without exceeding maximum load factor and rehashes the
container, i.e. puts the elements into appropriate buckets considering
that total number of buckets has changed
The important part is **at least*. I do not think you can rely on the implementation and be sure that it only uses less amount of memory. My guess is that regardless if you call reserve the memory usage for numElements will be similar.
Quoting the standard:
Sets the number of buckets to the number needed to accomodate at least
count elements without exceeding maximum load factor and rehashes the
container, i.e. puts the elements into appropriate buckets considering
that total number of buckets has changed
The important part is **at least*. I do not think you can rely on the implementation and be sure that it only uses less amount of memory. My guess is that regardless if you call reserve the memory usage for numElements will be similar.
answered Mar 27 at 8:33
Davide SpataroDavide Spataro
5,1031 gold badge13 silver badges32 bronze badges
5,1031 gold badge13 silver badges32 bronze badges
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55372630%2frole-of-stdunordered-setreserve-for-container-memory-requirement%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown