ClickHouse: How to delete on *AggregatingMergeTree tables from a materialized view The Next CEO of Stack OverflowHow make JOIN table in ClickHouse DB faster?Is Log a compressed table engine in ClickhouseWhy floating point representation are discuraged in clickhouse tables?Change column name in a table in ClickhouseHow to filter clickhouse table by array column contents?Is there a way to attach materialized view in ClickHouse?Clickhouse altering materialized view's selectClickhouse Kafka Engine: Materialized View benefitsIs it possible to delete old records from clickhouse table?How to avoid duplicates in clickhouse table?

Grabbing quick drinks

Does the Brexit deal have to be agreed by both Houses?

Term for the "extreme-extension" version of a straw man fallacy?

Text adventure game code

How to use tikz in fbox?

How to safely derail a train during transit?

How do I construct this japanese bowl?

Is a stroke of luck acceptable after a series of unfavorable events?

Why do professional authors make "consistency" mistakes? And how to avoid them?

How do we know the LHC results are robust?

What can we do to stop prior company from asking us questions?

Whats the best way to handle refactoring a big file?

Does it take more energy to get to Venus or to Mars?

Is it my responsibility to learn a new technology in my own time my employer wants to implement?

How to write the block matrix in LaTex?

I believe this to be a fraud - hired, then asked to cash check and send cash as Bitcoin

Opposite of a diet

Can a single photon have an energy density?

What makes a siege story/plot interesting?

Are there languages with no euphemisms?

Increase performance creating Mandelbrot set in python

How to Reset Passwords on Multiple Websites Easily?

Why doesn't a table tennis ball float on the surface? How do we calculate buoyancy here?

How do I solve this limit?



ClickHouse: How to delete on *AggregatingMergeTree tables from a materialized view



The Next CEO of Stack OverflowHow make JOIN table in ClickHouse DB faster?Is Log a compressed table engine in ClickhouseWhy floating point representation are discuraged in clickhouse tables?Change column name in a table in ClickhouseHow to filter clickhouse table by array column contents?Is there a way to attach materialized view in ClickHouse?Clickhouse altering materialized view's selectClickhouse Kafka Engine: Materialized View benefitsIs it possible to delete old records from clickhouse table?How to avoid duplicates in clickhouse table?










0















Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



Is this the expected behaviour? How to DELETE in such a schema?



I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d










share|improve this question


























    0















    Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



    However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



    Is this the expected behaviour? How to DELETE in such a schema?



    I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



    Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d










    share|improve this question
























      0












      0








      0








      Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



      However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



      Is this the expected behaviour? How to DELETE in such a schema?



      I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



      Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d










      share|improve this question














      Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



      However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



      Is this the expected behaviour? How to DELETE in such a schema?



      I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



      Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d







      clickhouse






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Mar 21 at 16:21









      xmarxmar

      484314




      484314






















          1 Answer
          1






          active

          oldest

          votes


















          1














          Mutations are not propagated to materialized views.



          The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



          For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



          So in general case, you need to refill the whole partition for your AggregatedMergeTree.



          I.e. something like (daily partitioning case):



           ALTER amt_table DROP PARTITION '2019-03-01';

          -- use same select as in your materialized view
          INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






          share|improve this answer























            Your Answer






            StackExchange.ifUsing("editor", function ()
            StackExchange.using("externalEditor", function ()
            StackExchange.using("snippets", function ()
            StackExchange.snippets.init();
            );
            );
            , "code-snippets");

            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "1"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );













            draft saved

            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55284961%2fclickhouse-how-to-delete-on-aggregatingmergetree-tables-from-a-materialized-vi%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            1














            Mutations are not propagated to materialized views.



            The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



            For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



            So in general case, you need to refill the whole partition for your AggregatedMergeTree.



            I.e. something like (daily partitioning case):



             ALTER amt_table DROP PARTITION '2019-03-01';

            -- use same select as in your materialized view
            INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






            share|improve this answer



























              1














              Mutations are not propagated to materialized views.



              The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



              For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



              So in general case, you need to refill the whole partition for your AggregatedMergeTree.



              I.e. something like (daily partitioning case):



               ALTER amt_table DROP PARTITION '2019-03-01';

              -- use same select as in your materialized view
              INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






              share|improve this answer

























                1












                1








                1







                Mutations are not propagated to materialized views.



                The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



                For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



                So in general case, you need to refill the whole partition for your AggregatedMergeTree.



                I.e. something like (daily partitioning case):



                 ALTER amt_table DROP PARTITION '2019-03-01';

                -- use same select as in your materialized view
                INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






                share|improve this answer













                Mutations are not propagated to materialized views.



                The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



                For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



                So in general case, you need to refill the whole partition for your AggregatedMergeTree.



                I.e. something like (daily partitioning case):



                 ALTER amt_table DROP PARTITION '2019-03-01';

                -- use same select as in your materialized view
                INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Mar 22 at 8:36









                filimonovfilimonov

                6991414




                6991414





























                    draft saved

                    draft discarded
















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid


                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.

                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55284961%2fclickhouse-how-to-delete-on-aggregatingmergetree-tables-from-a-materialized-vi%23new-answer', 'question_page');

                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Kamusi Yaliyomo Aina za kamusi | Muundo wa kamusi | Faida za kamusi | Dhima ya picha katika kamusi | Marejeo | Tazama pia | Viungo vya nje | UrambazajiKuhusu kamusiGo-SwahiliWiki-KamusiKamusi ya Kiswahili na Kiingerezakuihariri na kuongeza habari

                    Swift 4 - func physicsWorld not invoked on collision? The Next CEO of Stack OverflowHow to call Objective-C code from Swift#ifdef replacement in the Swift language@selector() in Swift?#pragma mark in Swift?Swift for loop: for index, element in array?dispatch_after - GCD in Swift?Swift Beta performance: sorting arraysSplit a String into an array in Swift?The use of Swift 3 @objc inference in Swift 4 mode is deprecated?How to optimize UITableViewCell, because my UITableView lags

                    Access current req object everywhere in Node.js ExpressWhy are global variables considered bad practice? (node.js)Using req & res across functionsHow do I get the path to the current script with Node.js?What is Node.js' Connect, Express and “middleware”?Node.js w/ express error handling in callbackHow to access the GET parameters after “?” in Express?Modify Node.js req object parametersAccess “app” variable inside of ExpressJS/ConnectJS middleware?Node.js Express app - request objectAngular Http Module considered middleware?Session variables in ExpressJSAdd properties to the req object in expressjs with Typescript