ClickHouse: How to delete on *AggregatingMergeTree tables from a materialized view The Next CEO of Stack OverflowHow make JOIN table in ClickHouse DB faster?Is Log a compressed table engine in ClickhouseWhy floating point representation are discuraged in clickhouse tables?Change column name in a table in ClickhouseHow to filter clickhouse table by array column contents?Is there a way to attach materialized view in ClickHouse?Clickhouse altering materialized view's selectClickhouse Kafka Engine: Materialized View benefitsIs it possible to delete old records from clickhouse table?How to avoid duplicates in clickhouse table?

Grabbing quick drinks

Does the Brexit deal have to be agreed by both Houses?

Term for the "extreme-extension" version of a straw man fallacy?

Text adventure game code

How to use tikz in fbox?

How to safely derail a train during transit?

How do I construct this japanese bowl?

Is a stroke of luck acceptable after a series of unfavorable events?

Why do professional authors make "consistency" mistakes? And how to avoid them?

How do we know the LHC results are robust?

What can we do to stop prior company from asking us questions?

Whats the best way to handle refactoring a big file?

Does it take more energy to get to Venus or to Mars?

Is it my responsibility to learn a new technology in my own time my employer wants to implement?

How to write the block matrix in LaTex?

I believe this to be a fraud - hired, then asked to cash check and send cash as Bitcoin

Opposite of a diet

Can a single photon have an energy density?

What makes a siege story/plot interesting?

Are there languages with no euphemisms?

Increase performance creating Mandelbrot set in python

How to Reset Passwords on Multiple Websites Easily?

Why doesn't a table tennis ball float on the surface? How do we calculate buoyancy here?

How do I solve this limit?



ClickHouse: How to delete on *AggregatingMergeTree tables from a materialized view



The Next CEO of Stack OverflowHow make JOIN table in ClickHouse DB faster?Is Log a compressed table engine in ClickhouseWhy floating point representation are discuraged in clickhouse tables?Change column name in a table in ClickhouseHow to filter clickhouse table by array column contents?Is there a way to attach materialized view in ClickHouse?Clickhouse altering materialized view's selectClickhouse Kafka Engine: Materialized View benefitsIs it possible to delete old records from clickhouse table?How to avoid duplicates in clickhouse table?










0















Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



Is this the expected behaviour? How to DELETE in such a schema?



I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d










share|improve this question


























    0















    Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



    However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



    Is this the expected behaviour? How to DELETE in such a schema?



    I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



    Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d










    share|improve this question
























      0












      0








      0








      Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



      However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



      Is this the expected behaviour? How to DELETE in such a schema?



      I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



      Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d










      share|improve this question














      Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).



      However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.



      Is this the expected behaviour? How to DELETE in such a schema?



      I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.



      Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d







      clickhouse






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Mar 21 at 16:21









      xmarxmar

      484314




      484314






















          1 Answer
          1






          active

          oldest

          votes


















          1














          Mutations are not propagated to materialized views.



          The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



          For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



          So in general case, you need to refill the whole partition for your AggregatedMergeTree.



          I.e. something like (daily partitioning case):



           ALTER amt_table DROP PARTITION '2019-03-01';

          -- use same select as in your materialized view
          INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






          share|improve this answer























            Your Answer






            StackExchange.ifUsing("editor", function ()
            StackExchange.using("externalEditor", function ()
            StackExchange.using("snippets", function ()
            StackExchange.snippets.init();
            );
            );
            , "code-snippets");

            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "1"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );













            draft saved

            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55284961%2fclickhouse-how-to-delete-on-aggregatingmergetree-tables-from-a-materialized-vi%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            1














            Mutations are not propagated to materialized views.



            The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



            For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



            So in general case, you need to refill the whole partition for your AggregatedMergeTree.



            I.e. something like (daily partitioning case):



             ALTER amt_table DROP PARTITION '2019-03-01';

            -- use same select as in your materialized view
            INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






            share|improve this answer



























              1














              Mutations are not propagated to materialized views.



              The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



              For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



              So in general case, you need to refill the whole partition for your AggregatedMergeTree.



              I.e. something like (daily partitioning case):



               ALTER amt_table DROP PARTITION '2019-03-01';

              -- use same select as in your materialized view
              INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






              share|improve this answer

























                1












                1








                1







                Mutations are not propagated to materialized views.



                The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



                For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



                So in general case, you need to refill the whole partition for your AggregatedMergeTree.



                I.e. something like (daily partitioning case):



                 ALTER amt_table DROP PARTITION '2019-03-01';

                -- use same select as in your materialized view
                INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';






                share|improve this answer













                Mutations are not propagated to materialized views.



                The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.



                For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.



                So in general case, you need to refill the whole partition for your AggregatedMergeTree.



                I.e. something like (daily partitioning case):



                 ALTER amt_table DROP PARTITION '2019-03-01';

                -- use same select as in your materialized view
                INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Mar 22 at 8:36









                filimonovfilimonov

                6991414




                6991414





























                    draft saved

                    draft discarded
















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid


                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.

                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55284961%2fclickhouse-how-to-delete-on-aggregatingmergetree-tables-from-a-materialized-vi%23new-answer', 'question_page');

                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Kamusi Yaliyomo Aina za kamusi | Muundo wa kamusi | Faida za kamusi | Dhima ya picha katika kamusi | Marejeo | Tazama pia | Viungo vya nje | UrambazajiKuhusu kamusiGo-SwahiliWiki-KamusiKamusi ya Kiswahili na Kiingerezakuihariri na kuongeza habari

                    SQL error code 1064 with creating Laravel foreign keysForeign key constraints: When to use ON UPDATE and ON DELETEDropping column with foreign key Laravel error: General error: 1025 Error on renameLaravel SQL Can't create tableLaravel Migration foreign key errorLaravel php artisan migrate:refresh giving a syntax errorSQLSTATE[42S01]: Base table or view already exists or Base table or view already exists: 1050 Tableerror in migrating laravel file to xampp serverSyntax error or access violation: 1064:syntax to use near 'unsigned not null, modelName varchar(191) not null, title varchar(191) not nLaravel cannot create new table field in mysqlLaravel 5.7:Last migration creates table but is not registered in the migration table

                    은진 송씨 목차 역사 본관 분파 인물 조선 왕실과의 인척 관계 집성촌 항렬자 인구 같이 보기 각주 둘러보기 메뉴은진 송씨세종실록 149권, 지리지 충청도 공주목 은진현