ClickHouse: How to delete on *AggregatingMergeTree tables from a materialized view The Next CEO of Stack OverflowHow make JOIN table in ClickHouse DB faster?Is Log a compressed table engine in ClickhouseWhy floating point representation are discuraged in clickhouse tables?Change column name in a table in ClickhouseHow to filter clickhouse table by array column contents?Is there a way to attach materialized view in ClickHouse?Clickhouse altering materialized view's selectClickhouse Kafka Engine: Materialized View benefitsIs it possible to delete old records from clickhouse table?How to avoid duplicates in clickhouse table?
Grabbing quick drinks
Does the Brexit deal have to be agreed by both Houses?
Term for the "extreme-extension" version of a straw man fallacy?
Text adventure game code
How to use tikz in fbox?
How to safely derail a train during transit?
How do I construct this japanese bowl?
Is a stroke of luck acceptable after a series of unfavorable events?
Why do professional authors make "consistency" mistakes? And how to avoid them?
How do we know the LHC results are robust?
What can we do to stop prior company from asking us questions?
Whats the best way to handle refactoring a big file?
Does it take more energy to get to Venus or to Mars?
Is it my responsibility to learn a new technology in my own time my employer wants to implement?
How to write the block matrix in LaTex?
I believe this to be a fraud - hired, then asked to cash check and send cash as Bitcoin
Opposite of a diet
Can a single photon have an energy density?
What makes a siege story/plot interesting?
Are there languages with no euphemisms?
Increase performance creating Mandelbrot set in python
How to Reset Passwords on Multiple Websites Easily?
Why doesn't a table tennis ball float on the surface? How do we calculate buoyancy here?
How do I solve this limit?
ClickHouse: How to delete on *AggregatingMergeTree tables from a materialized view
The Next CEO of Stack OverflowHow make JOIN table in ClickHouse DB faster?Is Log a compressed table engine in ClickhouseWhy floating point representation are discuraged in clickhouse tables?Change column name in a table in ClickhouseHow to filter clickhouse table by array column contents?Is there a way to attach materialized view in ClickHouse?Clickhouse altering materialized view's selectClickhouse Kafka Engine: Materialized View benefitsIs it possible to delete old records from clickhouse table?How to avoid duplicates in clickhouse table?
Having a structure where there is a base
table, then a materialized view base_mv
that aggregates sending the result TO
an AggregatedMergeTree
table base_agg_by_id
. Then we have a view over this final table. base_unique
. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).
However, if I delete from base
, I would expect the base_mv
would trigger the mutation and then act on it, and reflected on the base_agg_by_id
, but it doesn't.
Is this the expected behaviour? How to DELETE
in such a schema?
I've seen here that in MVs that keep data you can act on .inner
tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime)
), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'
.
Note. For the record, we have these tables in a replicated environment using Replicated*
engine: base_d
, base_agg_by_id_d
, base_unique_d
clickhouse
add a comment |
Having a structure where there is a base
table, then a materialized view base_mv
that aggregates sending the result TO
an AggregatedMergeTree
table base_agg_by_id
. Then we have a view over this final table. base_unique
. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).
However, if I delete from base
, I would expect the base_mv
would trigger the mutation and then act on it, and reflected on the base_agg_by_id
, but it doesn't.
Is this the expected behaviour? How to DELETE
in such a schema?
I've seen here that in MVs that keep data you can act on .inner
tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime)
), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'
.
Note. For the record, we have these tables in a replicated environment using Replicated*
engine: base_d
, base_agg_by_id_d
, base_unique_d
clickhouse
add a comment |
Having a structure where there is a base
table, then a materialized view base_mv
that aggregates sending the result TO
an AggregatedMergeTree
table base_agg_by_id
. Then we have a view over this final table. base_unique
. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).
However, if I delete from base
, I would expect the base_mv
would trigger the mutation and then act on it, and reflected on the base_agg_by_id
, but it doesn't.
Is this the expected behaviour? How to DELETE
in such a schema?
I've seen here that in MVs that keep data you can act on .inner
tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime)
), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'
.
Note. For the record, we have these tables in a replicated environment using Replicated*
engine: base_d
, base_agg_by_id_d
, base_unique_d
clickhouse
Having a structure where there is a base
table, then a materialized view base_mv
that aggregates sending the result TO
an AggregatedMergeTree
table base_agg_by_id
. Then we have a view over this final table. base_unique
. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).
However, if I delete from base
, I would expect the base_mv
would trigger the mutation and then act on it, and reflected on the base_agg_by_id
, but it doesn't.
Is this the expected behaviour? How to DELETE
in such a schema?
I've seen here that in MVs that keep data you can act on .inner
tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime)
), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'
.
Note. For the record, we have these tables in a replicated environment using Replicated*
engine: base_d
, base_agg_by_id_d
, base_unique_d
clickhouse
clickhouse
asked Mar 21 at 16:21
xmarxmar
484314
484314
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Mutations are not propagated to materialized views.
The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.
For example, let's say you're deleting one record from the table which references some userid
. And your materialized view contains uniqState( userid )
. Data structures used for calculating uniqState
don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.
So in general case, you need to refill the whole partition for your AggregatedMergeTree.
I.e. something like (daily partitioning case):
ALTER amt_table DROP PARTITION '2019-03-01';
-- use same select as in your materialized view
INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55284961%2fclickhouse-how-to-delete-on-aggregatingmergetree-tables-from-a-materialized-vi%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Mutations are not propagated to materialized views.
The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.
For example, let's say you're deleting one record from the table which references some userid
. And your materialized view contains uniqState( userid )
. Data structures used for calculating uniqState
don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.
So in general case, you need to refill the whole partition for your AggregatedMergeTree.
I.e. something like (daily partitioning case):
ALTER amt_table DROP PARTITION '2019-03-01';
-- use same select as in your materialized view
INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';
add a comment |
Mutations are not propagated to materialized views.
The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.
For example, let's say you're deleting one record from the table which references some userid
. And your materialized view contains uniqState( userid )
. Data structures used for calculating uniqState
don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.
So in general case, you need to refill the whole partition for your AggregatedMergeTree.
I.e. something like (daily partitioning case):
ALTER amt_table DROP PARTITION '2019-03-01';
-- use same select as in your materialized view
INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';
add a comment |
Mutations are not propagated to materialized views.
The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.
For example, let's say you're deleting one record from the table which references some userid
. And your materialized view contains uniqState( userid )
. Data structures used for calculating uniqState
don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.
So in general case, you need to refill the whole partition for your AggregatedMergeTree.
I.e. something like (daily partitioning case):
ALTER amt_table DROP PARTITION '2019-03-01';
-- use same select as in your materialized view
INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';
Mutations are not propagated to materialized views.
The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.
For example, let's say you're deleting one record from the table which references some userid
. And your materialized view contains uniqState( userid )
. Data structures used for calculating uniqState
don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.
So in general case, you need to refill the whole partition for your AggregatedMergeTree.
I.e. something like (daily partitioning case):
ALTER amt_table DROP PARTITION '2019-03-01';
-- use same select as in your materialized view
INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';
answered Mar 22 at 8:36
filimonovfilimonov
6991414
6991414
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55284961%2fclickhouse-how-to-delete-on-aggregatingmergetree-tables-from-a-materialized-vi%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown