Slow Postgres query using LIMITPostgreSQL query very slow with limit 1Extremely slow PostgreSQL query with ORDER and LIMIT clausespsql: FATAL: Ident authentication failed for user “postgres”postgres: upgrade a user to be a superuser?Getting error: Peer authentication failed for user “postgres”, when trying to get pgsql working with railsPostgreSQL - Query OptimizationSlow postgres text column queryPostgresql very slow query on indexed columnOptimising UUID Lookups in PostgresDjango-Rest-Framework - How to serialize queryset from an unrelated model as nested serializerHow to optimize a MAX SQL query with GROUP BY DATEWhy is our Postgres query performance slow?

Why did moving the mouse cursor cause Windows 95 to run more quickly?

How can I define a very large matrix efficiently?

PhD: When to quit and move on?

Is it acceptable that I plot a time-series figure with years increasing from right to left?

In the Seventh Seal why does Death let the chess game happen?

Do I need to be legally qualified to install a Hive smart thermostat?

What is this arch-and-tower near a road?

Do the 26 richest billionaires own as much wealth as the poorest 3.8 billion people?

Advice for making/keeping shredded chicken moist?

Should I cheat if the majority does it?

Creating patterns

Why would "dead languages" be the only languages that spells could be written in?

Minimizing medical costs with HSA

Isn't "Dave's protocol" good if only the database, and not the code, is leaked?

What is the addition in the re-released version of Avengers: Endgame?

How to travel between two stationary worlds in the least amount of time? (time dilation)

What is exact meaning of “ich wäre gern”?

Why did C++11 make std::string::data() add a null terminating character?

Boss has banned cycling to work because he thinks it's unsafe

How can solar sailed ships be protected from space debris?

Convenience stores in India

What are the differences of checking a self-signed certificate vs ignore it?

Can you use a weapon affected by Heat Metal each turn if you drop it in between?

Can a Time Lord survive with just one heart?



Slow Postgres query using LIMIT


PostgreSQL query very slow with limit 1Extremely slow PostgreSQL query with ORDER and LIMIT clausespsql: FATAL: Ident authentication failed for user “postgres”postgres: upgrade a user to be a superuser?Getting error: Peer authentication failed for user “postgres”, when trying to get pgsql working with railsPostgreSQL - Query OptimizationSlow postgres text column queryPostgresql very slow query on indexed columnOptimising UUID Lookups in PostgresDjango-Rest-Framework - How to serialize queryset from an unrelated model as nested serializerHow to optimize a MAX SQL query with GROUP BY DATEWhy is our Postgres query performance slow?






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;








4















I'm experiencing an issue similar to PostgreSQL query very slow with limit 1 and Extremely slow PostgreSQL query with ORDER and LIMIT clauses although in my case it doesn't matter if the LIMIT is 1, 5 or 500.



Basically, when I run a query generated by Django's ORM without a limit, the query takes half a second but with the limit (added for pagination) it takes 7 seconds.



The query that takes 7 seconds is:



SELECT "buildout_itemdescription"."product_code_id",
MIN("buildout_lineitem"."unit_price") AS "min_price"
FROM "buildout_lineitem"
INNER JOIN "buildout_itemdescription"
ON ("buildout_lineitem"."item_description_id" = "buildout_itemdescription"."id")
WHERE (("buildout_lineitem"."report_file_id" IN (154, 172, 155, 181, 174, 156, 157, 182, 175, 176, 183, 158, 177, 159, 179, 178, 164, 180, 367, 165, 173, 166, 167, 168, 368, 422, 370, 169, 1335, 1323, 161, 160, 162, 170, 171, 676, 151, 163, 980, 152, 369, 153, 963, 1718, 881, 617, 1759, 1780, 636, 1199, 1243, 947, 1163, 1422, 1009, 1407, 1035, 1241, 1077, 1271, 1111, 1130, 1489, 1507, 1555, 1600, 1619, 1663)
AND "buildout_lineitem"."unit_price" > 0
AND NOT ("buildout_itemdescription"."product_code_id" IS NULL)
AND "buildout_lineitem"."date" >= '2014-04-20'::date
AND "buildout_lineitem"."date" <= '2019-03-25'::date))
GROUP BY "buildout_itemdescription"."product_code_id"
ORDER BY "buildout_itemdescription"."product_code_id" LIMIT 5


And the other query is the same without the LIMIT 5.



Postgres is using very different plans for the two queries (HASH JOIN vs NESTED LOOP).



[

"Plan":
"Node Type": "Limit",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 1362.36,
"Plan Rows": 5,
"Plan Width": 36,
"Actual Startup Time": 7035.543,
"Actual Total Time": 7063.808,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "(min(buildout_lineitem.unit_price))"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 35365525.74,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 7035.541,
"Actual Total Time": 7063.804,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Nested Loop",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 0.86,
"Total Cost": 35362951.28,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 7035.220,
"Actual Total Time": 7062.420,
"Actual Rows": 10660,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": false,
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_itemdescription_product_code_id_084c51de",
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.43,
"Total Cost": 143970.67,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.074,
"Actual Total Time": 627.419,
"Actual Rows": 351127,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.id", "buildout_itemdescription.deleted", "buildout_itemdescription.created_on", "buildout_itemdescription.modified_on", "buildout_itemdescription.description", "buildout_itemdescription.category_1", "buildout_itemdescription.category_2", "buildout_itemdescription.category_3", "buildout_itemdescription.manufacturer_sku", "buildout_itemdescription.manufacturer_name", "buildout_itemdescription.distributor_sku", "buildout_itemdescription.supplier_id", "buildout_itemdescription.unit_of_measure", "buildout_itemdescription.quantity_in_unit_of_measure", "buildout_itemdescription.created_by_id", "buildout_itemdescription.modified_by_id", "buildout_itemdescription.product_code_id", "buildout_itemdescription.region_id"],
"Index Cond": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Index Recheck": 0,
"Shared Hit Blocks": 195375,
"Shared Read Blocks": 144994,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Index Scan",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_item_description_id_88254e09",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 35.00,
"Plan Rows": 9,
"Plan Width": 10,
"Actual Startup Time": 0.018,
"Actual Total Time": 0.018,
"Actual Rows": 0,
"Actual Loops": 351127,
"Output": ["buildout_lineitem.id", "buildout_lineitem.deleted", "buildout_lineitem.created_on", "buildout_lineitem.modified_on", "buildout_lineitem.date", "buildout_lineitem.month", "buildout_lineitem.quantity", "buildout_lineitem.unit_price", "buildout_lineitem.extended_sell", "buildout_lineitem.list_price", "buildout_lineitem.list_price_ext", "buildout_lineitem.agreed_price", "buildout_lineitem.agreed_price_ext", "buildout_lineitem.baseline_price", "buildout_lineitem.baseline_price_ext", "buildout_lineitem.item_class", "buildout_lineitem.user_created_by", "buildout_lineitem.sub_region", "buildout_lineitem.business_unit", "buildout_lineitem.created_by_id", "buildout_lineitem.item_description_id", "buildout_lineitem.modified_by_id", "buildout_lineitem.report_file_id"],
"Index Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date) AND (buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[])))",
"Rows Removed by Filter": 10,
"Shared Hit Blocks": 1799678,
"Shared Read Blocks": 1013783,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]
,
"Planning Time": 2.796,
"Triggers": [
],
"Execution Time": 7063.932

]


vs



[

"Plan":
"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 249781.35,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 561.755,
"Actual Total Time": 580.878,
"Actual Rows": 3771,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Sort",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 247206.89,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 561.741,
"Actual Total Time": 568.372,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Sort Key": ["buildout_itemdescription.product_code_id"],
"Sort Method": "external merge",
"Sort Space Used": 1624,
"Sort Space Type": "Disk",
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Hash Join",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 59518.27,
"Total Cost": 219295.38,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 429.997,
"Actual Total Time": 546.546,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": true,
"Hash Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Shared Hit Blocks": 5541,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3526,
"Temp Written Blocks": 3526,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_report_file_id_b56906e1",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 151740.64,
"Plan Rows": 332154,
"Plan Width": 10,
"Actual Startup Time": 0.063,
"Actual Total Time": 30.922,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_lineitem.unit_price", "buildout_lineitem.item_description_id"],
"Index Cond": "(buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[]))",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date))",
"Rows Removed by Filter": 6,
"Shared Hit Blocks": 1610,
"Shared Read Blocks": 991,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Startup Cost": 43050.17,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 429.373,
"Actual Total Time": 429.373,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Hash Buckets": 131072,
"Original Hash Buckets": 131072,
"Hash Batches": 16,
"Original Hash Batches": 16,
"Peak Memory Usage": 3495,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 3213,
"Plans": [

"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.00,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.021,
"Actual Total Time": 287.632,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Filter": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Filter": 301554,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]

]
,
"Planning Time": 2.965,
"Triggers": [
],
"Execution Time": 581.324

]


In Django, my code is:



 from rest_framework.pagination import LimitOffsetPagination


line_items = LineItem.objects.filter(
report_file__report=report,
unit_price__gt=0
).exclude(
item_description__product_code__isnull=True
)


 item_refs_aggregated = line_items.values(
'item_description__product_code_id'
).annotate(
min_price=Min('unit_price'),
).values(
"item_description__product_code",
"min_price",
).order_by(*ordering)


paginator = LimitOffsetPagination()
paginator.page_size = 10
result_page = paginator.paginate_queryset(
item_refs_aggregated,
request
)


The indexes on these tables are:
buildout_lineitem:



Indexes:
"buildout_lineitem_pkey" PRIMARY KEY, btree (id)
"buildout_lineitem_created_by_id_a61c52b7" btree (created_by_id)
"buildout_lineitem_item_description_id_88254e09" btree (item_description_id)
"buildout_lineitem_modified_by_id_8668530a" btree (modified_by_id)
"buildout_lineitem_report_file_id_b56906e1" btree (report_file_id)
Foreign-key constraints:
"buildout_lineitem_created_by_id_a61c52b7_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_item_description_id_88254e09_fk_buildout_" FOREIGN KEY (item_description_id) REFERENCES buildout_itemdescription(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_modified_by_id_8668530a_fk_auth_user_id" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_report_file_id_b56906e1_fk_buildout_" FOREIGN KEY (report_file_id) REFERENCES buildout_reportfile(id) DEFERRABLE INITIALLY DEFERRED


buildout_itemdescription:



Indexes:
"buildout_itemdescription_pkey" PRIMARY KEY, btree (id)
"buildout_itemdescription_created_by_id_32dc4fc9" btree (created_by_id)
"buildout_itemdescription_description_category_1_c_35f46dc6_idx" btree (description, category_1, category_2, category_3, manufacturer_sku, manufacturer_name, distributo
r_sku, supplier_id, unit_of_measure, quantity_in_unit_of_measure, region_id)
"buildout_itemdescription_distributor_name_id_ae72fd8a" btree (supplier_id)
"buildout_itemdescription_modified_by_id_58cf6676" btree (modified_by_id)
"buildout_itemdescription_product_code_id_084c51de" btree (product_code_id)
"buildout_itemdescription_region_id_c1682fff" btree (region_id)
Foreign-key constraints:
"buildout_itemdescrip_modified_by_id_58cf6676_fk_auth_user" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_product_code_id_084c51de_fk_buildout_" FOREIGN KEY (product_code_id) REFERENCES buildout_itemref(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_region_id_c1682fff_fk_buildout_" FOREIGN KEY (region_id) REFERENCES buildout_region(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_supplier_id_525b55c4_fk_buildout_" FOREIGN KEY (supplier_id) REFERENCES buildout_supplier(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescription_created_by_id_32dc4fc9_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED


Here are the relevant models:



LineItem



class LineItem(BaseModel):
'''
Field summary: date, month, unit_price, extended_sell,
list_price, list_price_ext, agreed_price, agreed_price_ext,
baseline_price, baseline_price_ext, item_class,
user_created_by, report_file, item_description
'''
# make a model regarding the file
# and include filename = db.Column(db.String())
# link to it as a foreign key

class Meta:
abstract = True

objects = LineItemManager()

date = models.DateField(
_("Date"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
month = models.DateField(
_("Month"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
quantity = models.IntegerField(
_("Qty"),
null=True,
blank=False,
)

unit_price = models.DecimalField(
_("Unit Price"),
max_digits=19,
decimal_places=2
)
extended_sell = models.DecimalField(
_("Ext Sell"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price = models.DecimalField(
_("List Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price_ext = models.DecimalField(
_("List Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price = models.DecimalField(
_("Agreed Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price_ext = models.DecimalField(
_("Agreed Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price = models.DecimalField(
_("Baseline Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price_ext = models.DecimalField(
_("Baseline Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
item_class = models.CharField(
_("Class"),
max_length=500,
null=True,
blank=True,
)

user_created_by = models.CharField(
_("User Created By"),
max_length=50,
null=True,
blank=True,
)

report_file = models.ForeignKey(
ReportFile,
verbose_name="Report File",
on_delete=models.CASCADE,
null=True,
blank=True,
)

item_description = models.ForeignKey(
"ItemDescription",
verbose_name=_("Item Desc"),
on_delete=models.CASCADE
)

sub_region = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

business_unit = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

# TODO: client property is untested
@property
def client(self):
return self.report_file.report.client

@property
def date_with_fallback(self):
if self.date:
return self.date
if self.month:
return self.month
return None

@client.setter
def client(self, value):
self.report_file.report.client = value

def __str__(self):
return f"self.date self.list_price"


ItemRef



class ItemRef(BaseModel):
identification_method = models.ForeignKey(
'IdentificationMethod',
verbose_name=_("Identification Method"),
on_delete=models.CASCADE
)

identification_values = JSONField()

def __str__(self):
return str(self.id)


ItemDescription



class ItemDescription(BaseModel):
'''
Field summary:
description, category_1, category_2, category_3
manufacturer_sku, manufacturer_name, distributor_sku,
supplier, unit_of_measure,
quantity_in_unit_of_measure, product_code, region
'''
description = models.TextField(
_("desc"),
blank=False,
null=True
)
category_1 = models.CharField(
_("Cat 1"),
max_length=500,
null=True,
blank=True,
)

category_2 = models.CharField(
_("Cat 2"),
max_length=500,
null=True,
blank=True,
)
category_3 = models.CharField(
_("Cat 3"),
max_length=500,
null=True,
blank=True,
)

manufacturer_sku = models.CharField(
_("Mfr SKU"),
max_length=500,
null=True,
)

manufacturer_name = models.CharField(
_("Mfr Name"),
max_length=200,
null=True,
)
distributor_sku = models.CharField(
_("Dist SKU"),
max_length=500,
null=True,
)

unit_of_measure = models.CharField(
_("UOM"),
max_length=50,
null=True,
)

quantity_in_unit_of_measure = models.IntegerField(
_("Qty In UOM"),
null=True,
blank=True,
)

product_code = models.ForeignKey(
ItemRef,
verbose_name=_("Product Code"),
on_delete=models.SET_NULL,
null=True,
blank=True,
)

supplier = models.ForeignKey(
Supplier,
verbose_name=_("Dist Name"),
on_delete=models.CASCADE
)

region = models.ForeignKey(
Region, verbose_name=_("Region"),
on_delete=models.CASCADE
)

def get_fields(self, fields):
# returns the selected fields as a list and the id
result = []
for field in fields:
value = getattr(self, field)
result.append(value)
return tuple(result), self.pk

def __str__(self):
return str(self.description)

class Meta:
index_together = [
[
'description', 'category_1', 'category_2',
'category_3', 'manufacturer_sku', 'manufacturer_name',
'distributor_sku', 'supplier', 'unit_of_measure',
'quantity_in_unit_of_measure', 'region'
]
]


I've tried running analyze on the tables in Postgres, but it didn't help. In the question I linked to, the answer was to add an index but I'm not sure what index I'd add here.










share|improve this question
























  • Can you show us the django queries you used?

    – Hybrid
    Mar 25 at 18:49











  • I added in a simplified version of my Django code

    – Zev
    Mar 25 at 18:56











  • Do you have any indexes?

    – schillingt
    Mar 25 at 18:59











  • Just what Django and Postgres are creating by default. The item description table has an index of all of its fields together for a different query. I can paste in what info postgres has about the indexes.

    – Zev
    Mar 25 at 19:07












  • An index for unit_price is conspicuously missing from lineitem; this is probably where I would start.

    – Endre Both
    Mar 25 at 19:31

















4















I'm experiencing an issue similar to PostgreSQL query very slow with limit 1 and Extremely slow PostgreSQL query with ORDER and LIMIT clauses although in my case it doesn't matter if the LIMIT is 1, 5 or 500.



Basically, when I run a query generated by Django's ORM without a limit, the query takes half a second but with the limit (added for pagination) it takes 7 seconds.



The query that takes 7 seconds is:



SELECT "buildout_itemdescription"."product_code_id",
MIN("buildout_lineitem"."unit_price") AS "min_price"
FROM "buildout_lineitem"
INNER JOIN "buildout_itemdescription"
ON ("buildout_lineitem"."item_description_id" = "buildout_itemdescription"."id")
WHERE (("buildout_lineitem"."report_file_id" IN (154, 172, 155, 181, 174, 156, 157, 182, 175, 176, 183, 158, 177, 159, 179, 178, 164, 180, 367, 165, 173, 166, 167, 168, 368, 422, 370, 169, 1335, 1323, 161, 160, 162, 170, 171, 676, 151, 163, 980, 152, 369, 153, 963, 1718, 881, 617, 1759, 1780, 636, 1199, 1243, 947, 1163, 1422, 1009, 1407, 1035, 1241, 1077, 1271, 1111, 1130, 1489, 1507, 1555, 1600, 1619, 1663)
AND "buildout_lineitem"."unit_price" > 0
AND NOT ("buildout_itemdescription"."product_code_id" IS NULL)
AND "buildout_lineitem"."date" >= '2014-04-20'::date
AND "buildout_lineitem"."date" <= '2019-03-25'::date))
GROUP BY "buildout_itemdescription"."product_code_id"
ORDER BY "buildout_itemdescription"."product_code_id" LIMIT 5


And the other query is the same without the LIMIT 5.



Postgres is using very different plans for the two queries (HASH JOIN vs NESTED LOOP).



[

"Plan":
"Node Type": "Limit",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 1362.36,
"Plan Rows": 5,
"Plan Width": 36,
"Actual Startup Time": 7035.543,
"Actual Total Time": 7063.808,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "(min(buildout_lineitem.unit_price))"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 35365525.74,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 7035.541,
"Actual Total Time": 7063.804,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Nested Loop",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 0.86,
"Total Cost": 35362951.28,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 7035.220,
"Actual Total Time": 7062.420,
"Actual Rows": 10660,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": false,
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_itemdescription_product_code_id_084c51de",
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.43,
"Total Cost": 143970.67,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.074,
"Actual Total Time": 627.419,
"Actual Rows": 351127,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.id", "buildout_itemdescription.deleted", "buildout_itemdescription.created_on", "buildout_itemdescription.modified_on", "buildout_itemdescription.description", "buildout_itemdescription.category_1", "buildout_itemdescription.category_2", "buildout_itemdescription.category_3", "buildout_itemdescription.manufacturer_sku", "buildout_itemdescription.manufacturer_name", "buildout_itemdescription.distributor_sku", "buildout_itemdescription.supplier_id", "buildout_itemdescription.unit_of_measure", "buildout_itemdescription.quantity_in_unit_of_measure", "buildout_itemdescription.created_by_id", "buildout_itemdescription.modified_by_id", "buildout_itemdescription.product_code_id", "buildout_itemdescription.region_id"],
"Index Cond": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Index Recheck": 0,
"Shared Hit Blocks": 195375,
"Shared Read Blocks": 144994,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Index Scan",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_item_description_id_88254e09",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 35.00,
"Plan Rows": 9,
"Plan Width": 10,
"Actual Startup Time": 0.018,
"Actual Total Time": 0.018,
"Actual Rows": 0,
"Actual Loops": 351127,
"Output": ["buildout_lineitem.id", "buildout_lineitem.deleted", "buildout_lineitem.created_on", "buildout_lineitem.modified_on", "buildout_lineitem.date", "buildout_lineitem.month", "buildout_lineitem.quantity", "buildout_lineitem.unit_price", "buildout_lineitem.extended_sell", "buildout_lineitem.list_price", "buildout_lineitem.list_price_ext", "buildout_lineitem.agreed_price", "buildout_lineitem.agreed_price_ext", "buildout_lineitem.baseline_price", "buildout_lineitem.baseline_price_ext", "buildout_lineitem.item_class", "buildout_lineitem.user_created_by", "buildout_lineitem.sub_region", "buildout_lineitem.business_unit", "buildout_lineitem.created_by_id", "buildout_lineitem.item_description_id", "buildout_lineitem.modified_by_id", "buildout_lineitem.report_file_id"],
"Index Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date) AND (buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[])))",
"Rows Removed by Filter": 10,
"Shared Hit Blocks": 1799678,
"Shared Read Blocks": 1013783,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]
,
"Planning Time": 2.796,
"Triggers": [
],
"Execution Time": 7063.932

]


vs



[

"Plan":
"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 249781.35,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 561.755,
"Actual Total Time": 580.878,
"Actual Rows": 3771,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Sort",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 247206.89,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 561.741,
"Actual Total Time": 568.372,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Sort Key": ["buildout_itemdescription.product_code_id"],
"Sort Method": "external merge",
"Sort Space Used": 1624,
"Sort Space Type": "Disk",
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Hash Join",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 59518.27,
"Total Cost": 219295.38,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 429.997,
"Actual Total Time": 546.546,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": true,
"Hash Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Shared Hit Blocks": 5541,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3526,
"Temp Written Blocks": 3526,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_report_file_id_b56906e1",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 151740.64,
"Plan Rows": 332154,
"Plan Width": 10,
"Actual Startup Time": 0.063,
"Actual Total Time": 30.922,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_lineitem.unit_price", "buildout_lineitem.item_description_id"],
"Index Cond": "(buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[]))",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date))",
"Rows Removed by Filter": 6,
"Shared Hit Blocks": 1610,
"Shared Read Blocks": 991,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Startup Cost": 43050.17,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 429.373,
"Actual Total Time": 429.373,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Hash Buckets": 131072,
"Original Hash Buckets": 131072,
"Hash Batches": 16,
"Original Hash Batches": 16,
"Peak Memory Usage": 3495,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 3213,
"Plans": [

"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.00,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.021,
"Actual Total Time": 287.632,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Filter": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Filter": 301554,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]

]
,
"Planning Time": 2.965,
"Triggers": [
],
"Execution Time": 581.324

]


In Django, my code is:



 from rest_framework.pagination import LimitOffsetPagination


line_items = LineItem.objects.filter(
report_file__report=report,
unit_price__gt=0
).exclude(
item_description__product_code__isnull=True
)


 item_refs_aggregated = line_items.values(
'item_description__product_code_id'
).annotate(
min_price=Min('unit_price'),
).values(
"item_description__product_code",
"min_price",
).order_by(*ordering)


paginator = LimitOffsetPagination()
paginator.page_size = 10
result_page = paginator.paginate_queryset(
item_refs_aggregated,
request
)


The indexes on these tables are:
buildout_lineitem:



Indexes:
"buildout_lineitem_pkey" PRIMARY KEY, btree (id)
"buildout_lineitem_created_by_id_a61c52b7" btree (created_by_id)
"buildout_lineitem_item_description_id_88254e09" btree (item_description_id)
"buildout_lineitem_modified_by_id_8668530a" btree (modified_by_id)
"buildout_lineitem_report_file_id_b56906e1" btree (report_file_id)
Foreign-key constraints:
"buildout_lineitem_created_by_id_a61c52b7_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_item_description_id_88254e09_fk_buildout_" FOREIGN KEY (item_description_id) REFERENCES buildout_itemdescription(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_modified_by_id_8668530a_fk_auth_user_id" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_report_file_id_b56906e1_fk_buildout_" FOREIGN KEY (report_file_id) REFERENCES buildout_reportfile(id) DEFERRABLE INITIALLY DEFERRED


buildout_itemdescription:



Indexes:
"buildout_itemdescription_pkey" PRIMARY KEY, btree (id)
"buildout_itemdescription_created_by_id_32dc4fc9" btree (created_by_id)
"buildout_itemdescription_description_category_1_c_35f46dc6_idx" btree (description, category_1, category_2, category_3, manufacturer_sku, manufacturer_name, distributo
r_sku, supplier_id, unit_of_measure, quantity_in_unit_of_measure, region_id)
"buildout_itemdescription_distributor_name_id_ae72fd8a" btree (supplier_id)
"buildout_itemdescription_modified_by_id_58cf6676" btree (modified_by_id)
"buildout_itemdescription_product_code_id_084c51de" btree (product_code_id)
"buildout_itemdescription_region_id_c1682fff" btree (region_id)
Foreign-key constraints:
"buildout_itemdescrip_modified_by_id_58cf6676_fk_auth_user" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_product_code_id_084c51de_fk_buildout_" FOREIGN KEY (product_code_id) REFERENCES buildout_itemref(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_region_id_c1682fff_fk_buildout_" FOREIGN KEY (region_id) REFERENCES buildout_region(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_supplier_id_525b55c4_fk_buildout_" FOREIGN KEY (supplier_id) REFERENCES buildout_supplier(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescription_created_by_id_32dc4fc9_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED


Here are the relevant models:



LineItem



class LineItem(BaseModel):
'''
Field summary: date, month, unit_price, extended_sell,
list_price, list_price_ext, agreed_price, agreed_price_ext,
baseline_price, baseline_price_ext, item_class,
user_created_by, report_file, item_description
'''
# make a model regarding the file
# and include filename = db.Column(db.String())
# link to it as a foreign key

class Meta:
abstract = True

objects = LineItemManager()

date = models.DateField(
_("Date"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
month = models.DateField(
_("Month"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
quantity = models.IntegerField(
_("Qty"),
null=True,
blank=False,
)

unit_price = models.DecimalField(
_("Unit Price"),
max_digits=19,
decimal_places=2
)
extended_sell = models.DecimalField(
_("Ext Sell"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price = models.DecimalField(
_("List Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price_ext = models.DecimalField(
_("List Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price = models.DecimalField(
_("Agreed Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price_ext = models.DecimalField(
_("Agreed Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price = models.DecimalField(
_("Baseline Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price_ext = models.DecimalField(
_("Baseline Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
item_class = models.CharField(
_("Class"),
max_length=500,
null=True,
blank=True,
)

user_created_by = models.CharField(
_("User Created By"),
max_length=50,
null=True,
blank=True,
)

report_file = models.ForeignKey(
ReportFile,
verbose_name="Report File",
on_delete=models.CASCADE,
null=True,
blank=True,
)

item_description = models.ForeignKey(
"ItemDescription",
verbose_name=_("Item Desc"),
on_delete=models.CASCADE
)

sub_region = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

business_unit = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

# TODO: client property is untested
@property
def client(self):
return self.report_file.report.client

@property
def date_with_fallback(self):
if self.date:
return self.date
if self.month:
return self.month
return None

@client.setter
def client(self, value):
self.report_file.report.client = value

def __str__(self):
return f"self.date self.list_price"


ItemRef



class ItemRef(BaseModel):
identification_method = models.ForeignKey(
'IdentificationMethod',
verbose_name=_("Identification Method"),
on_delete=models.CASCADE
)

identification_values = JSONField()

def __str__(self):
return str(self.id)


ItemDescription



class ItemDescription(BaseModel):
'''
Field summary:
description, category_1, category_2, category_3
manufacturer_sku, manufacturer_name, distributor_sku,
supplier, unit_of_measure,
quantity_in_unit_of_measure, product_code, region
'''
description = models.TextField(
_("desc"),
blank=False,
null=True
)
category_1 = models.CharField(
_("Cat 1"),
max_length=500,
null=True,
blank=True,
)

category_2 = models.CharField(
_("Cat 2"),
max_length=500,
null=True,
blank=True,
)
category_3 = models.CharField(
_("Cat 3"),
max_length=500,
null=True,
blank=True,
)

manufacturer_sku = models.CharField(
_("Mfr SKU"),
max_length=500,
null=True,
)

manufacturer_name = models.CharField(
_("Mfr Name"),
max_length=200,
null=True,
)
distributor_sku = models.CharField(
_("Dist SKU"),
max_length=500,
null=True,
)

unit_of_measure = models.CharField(
_("UOM"),
max_length=50,
null=True,
)

quantity_in_unit_of_measure = models.IntegerField(
_("Qty In UOM"),
null=True,
blank=True,
)

product_code = models.ForeignKey(
ItemRef,
verbose_name=_("Product Code"),
on_delete=models.SET_NULL,
null=True,
blank=True,
)

supplier = models.ForeignKey(
Supplier,
verbose_name=_("Dist Name"),
on_delete=models.CASCADE
)

region = models.ForeignKey(
Region, verbose_name=_("Region"),
on_delete=models.CASCADE
)

def get_fields(self, fields):
# returns the selected fields as a list and the id
result = []
for field in fields:
value = getattr(self, field)
result.append(value)
return tuple(result), self.pk

def __str__(self):
return str(self.description)

class Meta:
index_together = [
[
'description', 'category_1', 'category_2',
'category_3', 'manufacturer_sku', 'manufacturer_name',
'distributor_sku', 'supplier', 'unit_of_measure',
'quantity_in_unit_of_measure', 'region'
]
]


I've tried running analyze on the tables in Postgres, but it didn't help. In the question I linked to, the answer was to add an index but I'm not sure what index I'd add here.










share|improve this question
























  • Can you show us the django queries you used?

    – Hybrid
    Mar 25 at 18:49











  • I added in a simplified version of my Django code

    – Zev
    Mar 25 at 18:56











  • Do you have any indexes?

    – schillingt
    Mar 25 at 18:59











  • Just what Django and Postgres are creating by default. The item description table has an index of all of its fields together for a different query. I can paste in what info postgres has about the indexes.

    – Zev
    Mar 25 at 19:07












  • An index for unit_price is conspicuously missing from lineitem; this is probably where I would start.

    – Endre Both
    Mar 25 at 19:31













4












4








4


1






I'm experiencing an issue similar to PostgreSQL query very slow with limit 1 and Extremely slow PostgreSQL query with ORDER and LIMIT clauses although in my case it doesn't matter if the LIMIT is 1, 5 or 500.



Basically, when I run a query generated by Django's ORM without a limit, the query takes half a second but with the limit (added for pagination) it takes 7 seconds.



The query that takes 7 seconds is:



SELECT "buildout_itemdescription"."product_code_id",
MIN("buildout_lineitem"."unit_price") AS "min_price"
FROM "buildout_lineitem"
INNER JOIN "buildout_itemdescription"
ON ("buildout_lineitem"."item_description_id" = "buildout_itemdescription"."id")
WHERE (("buildout_lineitem"."report_file_id" IN (154, 172, 155, 181, 174, 156, 157, 182, 175, 176, 183, 158, 177, 159, 179, 178, 164, 180, 367, 165, 173, 166, 167, 168, 368, 422, 370, 169, 1335, 1323, 161, 160, 162, 170, 171, 676, 151, 163, 980, 152, 369, 153, 963, 1718, 881, 617, 1759, 1780, 636, 1199, 1243, 947, 1163, 1422, 1009, 1407, 1035, 1241, 1077, 1271, 1111, 1130, 1489, 1507, 1555, 1600, 1619, 1663)
AND "buildout_lineitem"."unit_price" > 0
AND NOT ("buildout_itemdescription"."product_code_id" IS NULL)
AND "buildout_lineitem"."date" >= '2014-04-20'::date
AND "buildout_lineitem"."date" <= '2019-03-25'::date))
GROUP BY "buildout_itemdescription"."product_code_id"
ORDER BY "buildout_itemdescription"."product_code_id" LIMIT 5


And the other query is the same without the LIMIT 5.



Postgres is using very different plans for the two queries (HASH JOIN vs NESTED LOOP).



[

"Plan":
"Node Type": "Limit",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 1362.36,
"Plan Rows": 5,
"Plan Width": 36,
"Actual Startup Time": 7035.543,
"Actual Total Time": 7063.808,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "(min(buildout_lineitem.unit_price))"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 35365525.74,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 7035.541,
"Actual Total Time": 7063.804,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Nested Loop",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 0.86,
"Total Cost": 35362951.28,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 7035.220,
"Actual Total Time": 7062.420,
"Actual Rows": 10660,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": false,
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_itemdescription_product_code_id_084c51de",
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.43,
"Total Cost": 143970.67,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.074,
"Actual Total Time": 627.419,
"Actual Rows": 351127,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.id", "buildout_itemdescription.deleted", "buildout_itemdescription.created_on", "buildout_itemdescription.modified_on", "buildout_itemdescription.description", "buildout_itemdescription.category_1", "buildout_itemdescription.category_2", "buildout_itemdescription.category_3", "buildout_itemdescription.manufacturer_sku", "buildout_itemdescription.manufacturer_name", "buildout_itemdescription.distributor_sku", "buildout_itemdescription.supplier_id", "buildout_itemdescription.unit_of_measure", "buildout_itemdescription.quantity_in_unit_of_measure", "buildout_itemdescription.created_by_id", "buildout_itemdescription.modified_by_id", "buildout_itemdescription.product_code_id", "buildout_itemdescription.region_id"],
"Index Cond": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Index Recheck": 0,
"Shared Hit Blocks": 195375,
"Shared Read Blocks": 144994,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Index Scan",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_item_description_id_88254e09",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 35.00,
"Plan Rows": 9,
"Plan Width": 10,
"Actual Startup Time": 0.018,
"Actual Total Time": 0.018,
"Actual Rows": 0,
"Actual Loops": 351127,
"Output": ["buildout_lineitem.id", "buildout_lineitem.deleted", "buildout_lineitem.created_on", "buildout_lineitem.modified_on", "buildout_lineitem.date", "buildout_lineitem.month", "buildout_lineitem.quantity", "buildout_lineitem.unit_price", "buildout_lineitem.extended_sell", "buildout_lineitem.list_price", "buildout_lineitem.list_price_ext", "buildout_lineitem.agreed_price", "buildout_lineitem.agreed_price_ext", "buildout_lineitem.baseline_price", "buildout_lineitem.baseline_price_ext", "buildout_lineitem.item_class", "buildout_lineitem.user_created_by", "buildout_lineitem.sub_region", "buildout_lineitem.business_unit", "buildout_lineitem.created_by_id", "buildout_lineitem.item_description_id", "buildout_lineitem.modified_by_id", "buildout_lineitem.report_file_id"],
"Index Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date) AND (buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[])))",
"Rows Removed by Filter": 10,
"Shared Hit Blocks": 1799678,
"Shared Read Blocks": 1013783,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]
,
"Planning Time": 2.796,
"Triggers": [
],
"Execution Time": 7063.932

]


vs



[

"Plan":
"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 249781.35,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 561.755,
"Actual Total Time": 580.878,
"Actual Rows": 3771,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Sort",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 247206.89,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 561.741,
"Actual Total Time": 568.372,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Sort Key": ["buildout_itemdescription.product_code_id"],
"Sort Method": "external merge",
"Sort Space Used": 1624,
"Sort Space Type": "Disk",
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Hash Join",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 59518.27,
"Total Cost": 219295.38,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 429.997,
"Actual Total Time": 546.546,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": true,
"Hash Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Shared Hit Blocks": 5541,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3526,
"Temp Written Blocks": 3526,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_report_file_id_b56906e1",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 151740.64,
"Plan Rows": 332154,
"Plan Width": 10,
"Actual Startup Time": 0.063,
"Actual Total Time": 30.922,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_lineitem.unit_price", "buildout_lineitem.item_description_id"],
"Index Cond": "(buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[]))",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date))",
"Rows Removed by Filter": 6,
"Shared Hit Blocks": 1610,
"Shared Read Blocks": 991,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Startup Cost": 43050.17,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 429.373,
"Actual Total Time": 429.373,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Hash Buckets": 131072,
"Original Hash Buckets": 131072,
"Hash Batches": 16,
"Original Hash Batches": 16,
"Peak Memory Usage": 3495,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 3213,
"Plans": [

"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.00,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.021,
"Actual Total Time": 287.632,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Filter": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Filter": 301554,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]

]
,
"Planning Time": 2.965,
"Triggers": [
],
"Execution Time": 581.324

]


In Django, my code is:



 from rest_framework.pagination import LimitOffsetPagination


line_items = LineItem.objects.filter(
report_file__report=report,
unit_price__gt=0
).exclude(
item_description__product_code__isnull=True
)


 item_refs_aggregated = line_items.values(
'item_description__product_code_id'
).annotate(
min_price=Min('unit_price'),
).values(
"item_description__product_code",
"min_price",
).order_by(*ordering)


paginator = LimitOffsetPagination()
paginator.page_size = 10
result_page = paginator.paginate_queryset(
item_refs_aggregated,
request
)


The indexes on these tables are:
buildout_lineitem:



Indexes:
"buildout_lineitem_pkey" PRIMARY KEY, btree (id)
"buildout_lineitem_created_by_id_a61c52b7" btree (created_by_id)
"buildout_lineitem_item_description_id_88254e09" btree (item_description_id)
"buildout_lineitem_modified_by_id_8668530a" btree (modified_by_id)
"buildout_lineitem_report_file_id_b56906e1" btree (report_file_id)
Foreign-key constraints:
"buildout_lineitem_created_by_id_a61c52b7_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_item_description_id_88254e09_fk_buildout_" FOREIGN KEY (item_description_id) REFERENCES buildout_itemdescription(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_modified_by_id_8668530a_fk_auth_user_id" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_report_file_id_b56906e1_fk_buildout_" FOREIGN KEY (report_file_id) REFERENCES buildout_reportfile(id) DEFERRABLE INITIALLY DEFERRED


buildout_itemdescription:



Indexes:
"buildout_itemdescription_pkey" PRIMARY KEY, btree (id)
"buildout_itemdescription_created_by_id_32dc4fc9" btree (created_by_id)
"buildout_itemdescription_description_category_1_c_35f46dc6_idx" btree (description, category_1, category_2, category_3, manufacturer_sku, manufacturer_name, distributo
r_sku, supplier_id, unit_of_measure, quantity_in_unit_of_measure, region_id)
"buildout_itemdescription_distributor_name_id_ae72fd8a" btree (supplier_id)
"buildout_itemdescription_modified_by_id_58cf6676" btree (modified_by_id)
"buildout_itemdescription_product_code_id_084c51de" btree (product_code_id)
"buildout_itemdescription_region_id_c1682fff" btree (region_id)
Foreign-key constraints:
"buildout_itemdescrip_modified_by_id_58cf6676_fk_auth_user" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_product_code_id_084c51de_fk_buildout_" FOREIGN KEY (product_code_id) REFERENCES buildout_itemref(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_region_id_c1682fff_fk_buildout_" FOREIGN KEY (region_id) REFERENCES buildout_region(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_supplier_id_525b55c4_fk_buildout_" FOREIGN KEY (supplier_id) REFERENCES buildout_supplier(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescription_created_by_id_32dc4fc9_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED


Here are the relevant models:



LineItem



class LineItem(BaseModel):
'''
Field summary: date, month, unit_price, extended_sell,
list_price, list_price_ext, agreed_price, agreed_price_ext,
baseline_price, baseline_price_ext, item_class,
user_created_by, report_file, item_description
'''
# make a model regarding the file
# and include filename = db.Column(db.String())
# link to it as a foreign key

class Meta:
abstract = True

objects = LineItemManager()

date = models.DateField(
_("Date"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
month = models.DateField(
_("Month"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
quantity = models.IntegerField(
_("Qty"),
null=True,
blank=False,
)

unit_price = models.DecimalField(
_("Unit Price"),
max_digits=19,
decimal_places=2
)
extended_sell = models.DecimalField(
_("Ext Sell"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price = models.DecimalField(
_("List Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price_ext = models.DecimalField(
_("List Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price = models.DecimalField(
_("Agreed Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price_ext = models.DecimalField(
_("Agreed Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price = models.DecimalField(
_("Baseline Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price_ext = models.DecimalField(
_("Baseline Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
item_class = models.CharField(
_("Class"),
max_length=500,
null=True,
blank=True,
)

user_created_by = models.CharField(
_("User Created By"),
max_length=50,
null=True,
blank=True,
)

report_file = models.ForeignKey(
ReportFile,
verbose_name="Report File",
on_delete=models.CASCADE,
null=True,
blank=True,
)

item_description = models.ForeignKey(
"ItemDescription",
verbose_name=_("Item Desc"),
on_delete=models.CASCADE
)

sub_region = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

business_unit = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

# TODO: client property is untested
@property
def client(self):
return self.report_file.report.client

@property
def date_with_fallback(self):
if self.date:
return self.date
if self.month:
return self.month
return None

@client.setter
def client(self, value):
self.report_file.report.client = value

def __str__(self):
return f"self.date self.list_price"


ItemRef



class ItemRef(BaseModel):
identification_method = models.ForeignKey(
'IdentificationMethod',
verbose_name=_("Identification Method"),
on_delete=models.CASCADE
)

identification_values = JSONField()

def __str__(self):
return str(self.id)


ItemDescription



class ItemDescription(BaseModel):
'''
Field summary:
description, category_1, category_2, category_3
manufacturer_sku, manufacturer_name, distributor_sku,
supplier, unit_of_measure,
quantity_in_unit_of_measure, product_code, region
'''
description = models.TextField(
_("desc"),
blank=False,
null=True
)
category_1 = models.CharField(
_("Cat 1"),
max_length=500,
null=True,
blank=True,
)

category_2 = models.CharField(
_("Cat 2"),
max_length=500,
null=True,
blank=True,
)
category_3 = models.CharField(
_("Cat 3"),
max_length=500,
null=True,
blank=True,
)

manufacturer_sku = models.CharField(
_("Mfr SKU"),
max_length=500,
null=True,
)

manufacturer_name = models.CharField(
_("Mfr Name"),
max_length=200,
null=True,
)
distributor_sku = models.CharField(
_("Dist SKU"),
max_length=500,
null=True,
)

unit_of_measure = models.CharField(
_("UOM"),
max_length=50,
null=True,
)

quantity_in_unit_of_measure = models.IntegerField(
_("Qty In UOM"),
null=True,
blank=True,
)

product_code = models.ForeignKey(
ItemRef,
verbose_name=_("Product Code"),
on_delete=models.SET_NULL,
null=True,
blank=True,
)

supplier = models.ForeignKey(
Supplier,
verbose_name=_("Dist Name"),
on_delete=models.CASCADE
)

region = models.ForeignKey(
Region, verbose_name=_("Region"),
on_delete=models.CASCADE
)

def get_fields(self, fields):
# returns the selected fields as a list and the id
result = []
for field in fields:
value = getattr(self, field)
result.append(value)
return tuple(result), self.pk

def __str__(self):
return str(self.description)

class Meta:
index_together = [
[
'description', 'category_1', 'category_2',
'category_3', 'manufacturer_sku', 'manufacturer_name',
'distributor_sku', 'supplier', 'unit_of_measure',
'quantity_in_unit_of_measure', 'region'
]
]


I've tried running analyze on the tables in Postgres, but it didn't help. In the question I linked to, the answer was to add an index but I'm not sure what index I'd add here.










share|improve this question
















I'm experiencing an issue similar to PostgreSQL query very slow with limit 1 and Extremely slow PostgreSQL query with ORDER and LIMIT clauses although in my case it doesn't matter if the LIMIT is 1, 5 or 500.



Basically, when I run a query generated by Django's ORM without a limit, the query takes half a second but with the limit (added for pagination) it takes 7 seconds.



The query that takes 7 seconds is:



SELECT "buildout_itemdescription"."product_code_id",
MIN("buildout_lineitem"."unit_price") AS "min_price"
FROM "buildout_lineitem"
INNER JOIN "buildout_itemdescription"
ON ("buildout_lineitem"."item_description_id" = "buildout_itemdescription"."id")
WHERE (("buildout_lineitem"."report_file_id" IN (154, 172, 155, 181, 174, 156, 157, 182, 175, 176, 183, 158, 177, 159, 179, 178, 164, 180, 367, 165, 173, 166, 167, 168, 368, 422, 370, 169, 1335, 1323, 161, 160, 162, 170, 171, 676, 151, 163, 980, 152, 369, 153, 963, 1718, 881, 617, 1759, 1780, 636, 1199, 1243, 947, 1163, 1422, 1009, 1407, 1035, 1241, 1077, 1271, 1111, 1130, 1489, 1507, 1555, 1600, 1619, 1663)
AND "buildout_lineitem"."unit_price" > 0
AND NOT ("buildout_itemdescription"."product_code_id" IS NULL)
AND "buildout_lineitem"."date" >= '2014-04-20'::date
AND "buildout_lineitem"."date" <= '2019-03-25'::date))
GROUP BY "buildout_itemdescription"."product_code_id"
ORDER BY "buildout_itemdescription"."product_code_id" LIMIT 5


And the other query is the same without the LIMIT 5.



Postgres is using very different plans for the two queries (HASH JOIN vs NESTED LOOP).



[

"Plan":
"Node Type": "Limit",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 1362.36,
"Plan Rows": 5,
"Plan Width": 36,
"Actual Startup Time": 7035.543,
"Actual Total Time": 7063.808,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "(min(buildout_lineitem.unit_price))"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 0.86,
"Total Cost": 35365525.74,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 7035.541,
"Actual Total Time": 7063.804,
"Actual Rows": 5,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Nested Loop",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 0.86,
"Total Cost": 35362951.28,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 7035.220,
"Actual Total Time": 7062.420,
"Actual Rows": 10660,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": false,
"Shared Hit Blocks": 1995053,
"Shared Read Blocks": 1158777,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_itemdescription_product_code_id_084c51de",
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.43,
"Total Cost": 143970.67,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.074,
"Actual Total Time": 627.419,
"Actual Rows": 351127,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.id", "buildout_itemdescription.deleted", "buildout_itemdescription.created_on", "buildout_itemdescription.modified_on", "buildout_itemdescription.description", "buildout_itemdescription.category_1", "buildout_itemdescription.category_2", "buildout_itemdescription.category_3", "buildout_itemdescription.manufacturer_sku", "buildout_itemdescription.manufacturer_name", "buildout_itemdescription.distributor_sku", "buildout_itemdescription.supplier_id", "buildout_itemdescription.unit_of_measure", "buildout_itemdescription.quantity_in_unit_of_measure", "buildout_itemdescription.created_by_id", "buildout_itemdescription.modified_by_id", "buildout_itemdescription.product_code_id", "buildout_itemdescription.region_id"],
"Index Cond": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Index Recheck": 0,
"Shared Hit Blocks": 195375,
"Shared Read Blocks": 144994,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Index Scan",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_item_description_id_88254e09",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 35.00,
"Plan Rows": 9,
"Plan Width": 10,
"Actual Startup Time": 0.018,
"Actual Total Time": 0.018,
"Actual Rows": 0,
"Actual Loops": 351127,
"Output": ["buildout_lineitem.id", "buildout_lineitem.deleted", "buildout_lineitem.created_on", "buildout_lineitem.modified_on", "buildout_lineitem.date", "buildout_lineitem.month", "buildout_lineitem.quantity", "buildout_lineitem.unit_price", "buildout_lineitem.extended_sell", "buildout_lineitem.list_price", "buildout_lineitem.list_price_ext", "buildout_lineitem.agreed_price", "buildout_lineitem.agreed_price_ext", "buildout_lineitem.baseline_price", "buildout_lineitem.baseline_price_ext", "buildout_lineitem.item_class", "buildout_lineitem.user_created_by", "buildout_lineitem.sub_region", "buildout_lineitem.business_unit", "buildout_lineitem.created_by_id", "buildout_lineitem.item_description_id", "buildout_lineitem.modified_by_id", "buildout_lineitem.report_file_id"],
"Index Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date) AND (buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[])))",
"Rows Removed by Filter": 10,
"Shared Hit Blocks": 1799678,
"Shared Read Blocks": 1013783,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]
,
"Planning Time": 2.796,
"Triggers": [
],
"Execution Time": 7063.932

]


vs



[

"Plan":
"Node Type": "Aggregate",
"Strategy": "Sorted",
"Partial Mode": "Simple",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 249781.35,
"Plan Rows": 129877,
"Plan Width": 36,
"Actual Startup Time": 561.755,
"Actual Total Time": 580.878,
"Actual Rows": 3771,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "min(buildout_lineitem.unit_price)"],
"Group Key": ["buildout_itemdescription.product_code_id"],
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Sort",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Startup Cost": 246569.04,
"Total Cost": 247206.89,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 561.741,
"Actual Total Time": 568.372,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Sort Key": ["buildout_itemdescription.product_code_id"],
"Sort Method": "external merge",
"Sort Space Used": 1624,
"Sort Space Type": "Disk",
"Shared Hit Blocks": 5544,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3729,
"Temp Written Blocks": 3730,
"Plans": [

"Node Type": "Hash Join",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Join Type": "Inner",
"Startup Cost": 59518.27,
"Total Cost": 219295.38,
"Plan Rows": 255139,
"Plan Width": 10,
"Actual Startup Time": 429.997,
"Actual Total Time": 546.546,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_lineitem.unit_price"],
"Inner Unique": true,
"Hash Cond": "(buildout_lineitem.item_description_id = buildout_itemdescription.id)",
"Shared Hit Blocks": 5541,
"Shared Read Blocks": 27043,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 3526,
"Temp Written Blocks": 3526,
"Plans": [

"Node Type": "Index Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Scan Direction": "Forward",
"Index Name": "buildout_lineitem_report_file_id_b56906e1",
"Relation Name": "buildout_lineitem",
"Schema": "public",
"Alias": "buildout_lineitem",
"Startup Cost": 0.43,
"Total Cost": 151740.64,
"Plan Rows": 332154,
"Plan Width": 10,
"Actual Startup Time": 0.063,
"Actual Total Time": 30.922,
"Actual Rows": 79253,
"Actual Loops": 1,
"Output": ["buildout_lineitem.unit_price", "buildout_lineitem.item_description_id"],
"Index Cond": "(buildout_lineitem.report_file_id = ANY ('154,172,155,181,174,156,157,182,175,176,183,158,177,159,179,178,164,180,367,165,173,166,167,168,368,422,370,169,1335,1323,161,160,162,170,171,676,151,163,980,152,369,153,963,1718,881,617,1759,1780,636,1199,1243,947,1163,1422,1009,1407,1035,1241,1077,1271,1111,1130,1489,1507,1555,1600,1619,1663'::integer[]))",
"Rows Removed by Index Recheck": 0,
"Filter": "((buildout_lineitem.unit_price > '0'::numeric) AND (buildout_lineitem.date >= '2014-04-20'::date) AND (buildout_lineitem.date <= '2019-03-25'::date))",
"Rows Removed by Filter": 6,
"Shared Hit Blocks": 1610,
"Shared Read Blocks": 991,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0
,

"Node Type": "Hash",
"Parent Relationship": "Inner",
"Parallel Aware": false,
"Startup Cost": 43050.17,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 429.373,
"Actual Total Time": 429.373,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Hash Buckets": 131072,
"Original Hash Buckets": 131072,
"Hash Batches": 16,
"Original Hash Batches": 16,
"Peak Memory Usage": 3495,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 3213,
"Plans": [

"Node Type": "Seq Scan",
"Parent Relationship": "Outer",
"Parallel Aware": false,
"Relation Name": "buildout_itemdescription",
"Schema": "public",
"Alias": "buildout_itemdescription",
"Startup Cost": 0.00,
"Total Cost": 43050.17,
"Plan Rows": 1003733,
"Plan Width": 8,
"Actual Startup Time": 0.021,
"Actual Total Time": 287.632,
"Actual Rows": 1005163,
"Actual Loops": 1,
"Output": ["buildout_itemdescription.product_code_id", "buildout_itemdescription.id"],
"Filter": "(buildout_itemdescription.product_code_id IS NOT NULL)",
"Rows Removed by Filter": 301554,
"Shared Hit Blocks": 3931,
"Shared Read Blocks": 26052,
"Shared Dirtied Blocks": 0,
"Shared Written Blocks": 0,
"Local Hit Blocks": 0,
"Local Read Blocks": 0,
"Local Dirtied Blocks": 0,
"Local Written Blocks": 0,
"Temp Read Blocks": 0,
"Temp Written Blocks": 0

]

]

]

]
,
"Planning Time": 2.965,
"Triggers": [
],
"Execution Time": 581.324

]


In Django, my code is:



 from rest_framework.pagination import LimitOffsetPagination


line_items = LineItem.objects.filter(
report_file__report=report,
unit_price__gt=0
).exclude(
item_description__product_code__isnull=True
)


 item_refs_aggregated = line_items.values(
'item_description__product_code_id'
).annotate(
min_price=Min('unit_price'),
).values(
"item_description__product_code",
"min_price",
).order_by(*ordering)


paginator = LimitOffsetPagination()
paginator.page_size = 10
result_page = paginator.paginate_queryset(
item_refs_aggregated,
request
)


The indexes on these tables are:
buildout_lineitem:



Indexes:
"buildout_lineitem_pkey" PRIMARY KEY, btree (id)
"buildout_lineitem_created_by_id_a61c52b7" btree (created_by_id)
"buildout_lineitem_item_description_id_88254e09" btree (item_description_id)
"buildout_lineitem_modified_by_id_8668530a" btree (modified_by_id)
"buildout_lineitem_report_file_id_b56906e1" btree (report_file_id)
Foreign-key constraints:
"buildout_lineitem_created_by_id_a61c52b7_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_item_description_id_88254e09_fk_buildout_" FOREIGN KEY (item_description_id) REFERENCES buildout_itemdescription(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_modified_by_id_8668530a_fk_auth_user_id" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_lineitem_report_file_id_b56906e1_fk_buildout_" FOREIGN KEY (report_file_id) REFERENCES buildout_reportfile(id) DEFERRABLE INITIALLY DEFERRED


buildout_itemdescription:



Indexes:
"buildout_itemdescription_pkey" PRIMARY KEY, btree (id)
"buildout_itemdescription_created_by_id_32dc4fc9" btree (created_by_id)
"buildout_itemdescription_description_category_1_c_35f46dc6_idx" btree (description, category_1, category_2, category_3, manufacturer_sku, manufacturer_name, distributo
r_sku, supplier_id, unit_of_measure, quantity_in_unit_of_measure, region_id)
"buildout_itemdescription_distributor_name_id_ae72fd8a" btree (supplier_id)
"buildout_itemdescription_modified_by_id_58cf6676" btree (modified_by_id)
"buildout_itemdescription_product_code_id_084c51de" btree (product_code_id)
"buildout_itemdescription_region_id_c1682fff" btree (region_id)
Foreign-key constraints:
"buildout_itemdescrip_modified_by_id_58cf6676_fk_auth_user" FOREIGN KEY (modified_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_product_code_id_084c51de_fk_buildout_" FOREIGN KEY (product_code_id) REFERENCES buildout_itemref(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_region_id_c1682fff_fk_buildout_" FOREIGN KEY (region_id) REFERENCES buildout_region(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescrip_supplier_id_525b55c4_fk_buildout_" FOREIGN KEY (supplier_id) REFERENCES buildout_supplier(id) DEFERRABLE INITIALLY DEFERRED
"buildout_itemdescription_created_by_id_32dc4fc9_fk_auth_user_id" FOREIGN KEY (created_by_id) REFERENCES auth_user(id) DEFERRABLE INITIALLY DEFERRED


Here are the relevant models:



LineItem



class LineItem(BaseModel):
'''
Field summary: date, month, unit_price, extended_sell,
list_price, list_price_ext, agreed_price, agreed_price_ext,
baseline_price, baseline_price_ext, item_class,
user_created_by, report_file, item_description
'''
# make a model regarding the file
# and include filename = db.Column(db.String())
# link to it as a foreign key

class Meta:
abstract = True

objects = LineItemManager()

date = models.DateField(
_("Date"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
month = models.DateField(
_("Month"),
auto_now=False,
auto_now_add=False,
blank=False,
null=True
)
quantity = models.IntegerField(
_("Qty"),
null=True,
blank=False,
)

unit_price = models.DecimalField(
_("Unit Price"),
max_digits=19,
decimal_places=2
)
extended_sell = models.DecimalField(
_("Ext Sell"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price = models.DecimalField(
_("List Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
list_price_ext = models.DecimalField(
_("List Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price = models.DecimalField(
_("Agreed Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
agreed_price_ext = models.DecimalField(
_("Agreed Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price = models.DecimalField(
_("Baseline Price"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
baseline_price_ext = models.DecimalField(
_("Baseline Price Ext"),
null=True,
blank=True,
max_digits=19,
decimal_places=2,
)
item_class = models.CharField(
_("Class"),
max_length=500,
null=True,
blank=True,
)

user_created_by = models.CharField(
_("User Created By"),
max_length=50,
null=True,
blank=True,
)

report_file = models.ForeignKey(
ReportFile,
verbose_name="Report File",
on_delete=models.CASCADE,
null=True,
blank=True,
)

item_description = models.ForeignKey(
"ItemDescription",
verbose_name=_("Item Desc"),
on_delete=models.CASCADE
)

sub_region = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

business_unit = models.CharField(
_("Sub Region"),
max_length=500,
null=True,
blank=True,
)

# TODO: client property is untested
@property
def client(self):
return self.report_file.report.client

@property
def date_with_fallback(self):
if self.date:
return self.date
if self.month:
return self.month
return None

@client.setter
def client(self, value):
self.report_file.report.client = value

def __str__(self):
return f"self.date self.list_price"


ItemRef



class ItemRef(BaseModel):
identification_method = models.ForeignKey(
'IdentificationMethod',
verbose_name=_("Identification Method"),
on_delete=models.CASCADE
)

identification_values = JSONField()

def __str__(self):
return str(self.id)


ItemDescription



class ItemDescription(BaseModel):
'''
Field summary:
description, category_1, category_2, category_3
manufacturer_sku, manufacturer_name, distributor_sku,
supplier, unit_of_measure,
quantity_in_unit_of_measure, product_code, region
'''
description = models.TextField(
_("desc"),
blank=False,
null=True
)
category_1 = models.CharField(
_("Cat 1"),
max_length=500,
null=True,
blank=True,
)

category_2 = models.CharField(
_("Cat 2"),
max_length=500,
null=True,
blank=True,
)
category_3 = models.CharField(
_("Cat 3"),
max_length=500,
null=True,
blank=True,
)

manufacturer_sku = models.CharField(
_("Mfr SKU"),
max_length=500,
null=True,
)

manufacturer_name = models.CharField(
_("Mfr Name"),
max_length=200,
null=True,
)
distributor_sku = models.CharField(
_("Dist SKU"),
max_length=500,
null=True,
)

unit_of_measure = models.CharField(
_("UOM"),
max_length=50,
null=True,
)

quantity_in_unit_of_measure = models.IntegerField(
_("Qty In UOM"),
null=True,
blank=True,
)

product_code = models.ForeignKey(
ItemRef,
verbose_name=_("Product Code"),
on_delete=models.SET_NULL,
null=True,
blank=True,
)

supplier = models.ForeignKey(
Supplier,
verbose_name=_("Dist Name"),
on_delete=models.CASCADE
)

region = models.ForeignKey(
Region, verbose_name=_("Region"),
on_delete=models.CASCADE
)

def get_fields(self, fields):
# returns the selected fields as a list and the id
result = []
for field in fields:
value = getattr(self, field)
result.append(value)
return tuple(result), self.pk

def __str__(self):
return str(self.description)

class Meta:
index_together = [
[
'description', 'category_1', 'category_2',
'category_3', 'manufacturer_sku', 'manufacturer_name',
'distributor_sku', 'supplier', 'unit_of_measure',
'quantity_in_unit_of_measure', 'region'
]
]


I've tried running analyze on the tables in Postgres, but it didn't help. In the question I linked to, the answer was to add an index but I'm not sure what index I'd add here.







postgresql database-performance sql-limit






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 26 at 12:26









Endre Both

3,1411 gold badge13 silver badges22 bronze badges




3,1411 gold badge13 silver badges22 bronze badges










asked Mar 25 at 18:48









ZevZev

2,2711 gold badge12 silver badges28 bronze badges




2,2711 gold badge12 silver badges28 bronze badges












  • Can you show us the django queries you used?

    – Hybrid
    Mar 25 at 18:49











  • I added in a simplified version of my Django code

    – Zev
    Mar 25 at 18:56











  • Do you have any indexes?

    – schillingt
    Mar 25 at 18:59











  • Just what Django and Postgres are creating by default. The item description table has an index of all of its fields together for a different query. I can paste in what info postgres has about the indexes.

    – Zev
    Mar 25 at 19:07












  • An index for unit_price is conspicuously missing from lineitem; this is probably where I would start.

    – Endre Both
    Mar 25 at 19:31

















  • Can you show us the django queries you used?

    – Hybrid
    Mar 25 at 18:49











  • I added in a simplified version of my Django code

    – Zev
    Mar 25 at 18:56











  • Do you have any indexes?

    – schillingt
    Mar 25 at 18:59











  • Just what Django and Postgres are creating by default. The item description table has an index of all of its fields together for a different query. I can paste in what info postgres has about the indexes.

    – Zev
    Mar 25 at 19:07












  • An index for unit_price is conspicuously missing from lineitem; this is probably where I would start.

    – Endre Both
    Mar 25 at 19:31
















Can you show us the django queries you used?

– Hybrid
Mar 25 at 18:49





Can you show us the django queries you used?

– Hybrid
Mar 25 at 18:49













I added in a simplified version of my Django code

– Zev
Mar 25 at 18:56





I added in a simplified version of my Django code

– Zev
Mar 25 at 18:56













Do you have any indexes?

– schillingt
Mar 25 at 18:59





Do you have any indexes?

– schillingt
Mar 25 at 18:59













Just what Django and Postgres are creating by default. The item description table has an index of all of its fields together for a different query. I can paste in what info postgres has about the indexes.

– Zev
Mar 25 at 19:07






Just what Django and Postgres are creating by default. The item description table has an index of all of its fields together for a different query. I can paste in what info postgres has about the indexes.

– Zev
Mar 25 at 19:07














An index for unit_price is conspicuously missing from lineitem; this is probably where I would start.

– Endre Both
Mar 25 at 19:31





An index for unit_price is conspicuously missing from lineitem; this is probably where I would start.

– Endre Both
Mar 25 at 19:31












1 Answer
1






active

oldest

votes


















2














PostgreSQL mistakenly thinks that it can be fast by scanning the rows in product_code_id order until it has found the first one satisfying all conditions, but it is not aware that the first matching row is not close to the beginning. So it ends up executing more of the nested loop join than it bargained for.



Removing that index would speed up the query, but if that is not an option, try using



...
ORDER BY buildout_itemdescription.product_code_id + 0
LIMIT 5


Then PostgreSQL cannot use the index.






share|improve this answer

























  • Thanks, I think you definitely have the explanation correct. Unfortunately, when I manually test this query against my database it reverts to the slower method.

    – Zev
    Mar 26 at 13:27











  • I have changed the answer with another idea.

    – Laurenz Albe
    Mar 26 at 13:37











  • Thanks, that works! In the ORM, it was ordering = [F('item_description__product_code_id') + 0]

    – Zev
    Mar 26 at 13:59












  • Following your answer is still giving me the best performance but some db tuning for SSD has helped as well amplitude.engineering/…

    – Zev
    Mar 27 at 18:50











  • Sure. Telling your database about your hardware is a good idea. I would also increase effective_io_concurrency.

    – Laurenz Albe
    Mar 27 at 18:53










Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55344598%2fslow-postgres-query-using-limit%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









2














PostgreSQL mistakenly thinks that it can be fast by scanning the rows in product_code_id order until it has found the first one satisfying all conditions, but it is not aware that the first matching row is not close to the beginning. So it ends up executing more of the nested loop join than it bargained for.



Removing that index would speed up the query, but if that is not an option, try using



...
ORDER BY buildout_itemdescription.product_code_id + 0
LIMIT 5


Then PostgreSQL cannot use the index.






share|improve this answer

























  • Thanks, I think you definitely have the explanation correct. Unfortunately, when I manually test this query against my database it reverts to the slower method.

    – Zev
    Mar 26 at 13:27











  • I have changed the answer with another idea.

    – Laurenz Albe
    Mar 26 at 13:37











  • Thanks, that works! In the ORM, it was ordering = [F('item_description__product_code_id') + 0]

    – Zev
    Mar 26 at 13:59












  • Following your answer is still giving me the best performance but some db tuning for SSD has helped as well amplitude.engineering/…

    – Zev
    Mar 27 at 18:50











  • Sure. Telling your database about your hardware is a good idea. I would also increase effective_io_concurrency.

    – Laurenz Albe
    Mar 27 at 18:53















2














PostgreSQL mistakenly thinks that it can be fast by scanning the rows in product_code_id order until it has found the first one satisfying all conditions, but it is not aware that the first matching row is not close to the beginning. So it ends up executing more of the nested loop join than it bargained for.



Removing that index would speed up the query, but if that is not an option, try using



...
ORDER BY buildout_itemdescription.product_code_id + 0
LIMIT 5


Then PostgreSQL cannot use the index.






share|improve this answer

























  • Thanks, I think you definitely have the explanation correct. Unfortunately, when I manually test this query against my database it reverts to the slower method.

    – Zev
    Mar 26 at 13:27











  • I have changed the answer with another idea.

    – Laurenz Albe
    Mar 26 at 13:37











  • Thanks, that works! In the ORM, it was ordering = [F('item_description__product_code_id') + 0]

    – Zev
    Mar 26 at 13:59












  • Following your answer is still giving me the best performance but some db tuning for SSD has helped as well amplitude.engineering/…

    – Zev
    Mar 27 at 18:50











  • Sure. Telling your database about your hardware is a good idea. I would also increase effective_io_concurrency.

    – Laurenz Albe
    Mar 27 at 18:53













2












2








2







PostgreSQL mistakenly thinks that it can be fast by scanning the rows in product_code_id order until it has found the first one satisfying all conditions, but it is not aware that the first matching row is not close to the beginning. So it ends up executing more of the nested loop join than it bargained for.



Removing that index would speed up the query, but if that is not an option, try using



...
ORDER BY buildout_itemdescription.product_code_id + 0
LIMIT 5


Then PostgreSQL cannot use the index.






share|improve this answer















PostgreSQL mistakenly thinks that it can be fast by scanning the rows in product_code_id order until it has found the first one satisfying all conditions, but it is not aware that the first matching row is not close to the beginning. So it ends up executing more of the nested loop join than it bargained for.



Removing that index would speed up the query, but if that is not an option, try using



...
ORDER BY buildout_itemdescription.product_code_id + 0
LIMIT 5


Then PostgreSQL cannot use the index.







share|improve this answer














share|improve this answer



share|improve this answer








edited Mar 26 at 13:37

























answered Mar 25 at 21:28









Laurenz AlbeLaurenz Albe

58.7k11 gold badges42 silver badges63 bronze badges




58.7k11 gold badges42 silver badges63 bronze badges












  • Thanks, I think you definitely have the explanation correct. Unfortunately, when I manually test this query against my database it reverts to the slower method.

    – Zev
    Mar 26 at 13:27











  • I have changed the answer with another idea.

    – Laurenz Albe
    Mar 26 at 13:37











  • Thanks, that works! In the ORM, it was ordering = [F('item_description__product_code_id') + 0]

    – Zev
    Mar 26 at 13:59












  • Following your answer is still giving me the best performance but some db tuning for SSD has helped as well amplitude.engineering/…

    – Zev
    Mar 27 at 18:50











  • Sure. Telling your database about your hardware is a good idea. I would also increase effective_io_concurrency.

    – Laurenz Albe
    Mar 27 at 18:53

















  • Thanks, I think you definitely have the explanation correct. Unfortunately, when I manually test this query against my database it reverts to the slower method.

    – Zev
    Mar 26 at 13:27











  • I have changed the answer with another idea.

    – Laurenz Albe
    Mar 26 at 13:37











  • Thanks, that works! In the ORM, it was ordering = [F('item_description__product_code_id') + 0]

    – Zev
    Mar 26 at 13:59












  • Following your answer is still giving me the best performance but some db tuning for SSD has helped as well amplitude.engineering/…

    – Zev
    Mar 27 at 18:50











  • Sure. Telling your database about your hardware is a good idea. I would also increase effective_io_concurrency.

    – Laurenz Albe
    Mar 27 at 18:53
















Thanks, I think you definitely have the explanation correct. Unfortunately, when I manually test this query against my database it reverts to the slower method.

– Zev
Mar 26 at 13:27





Thanks, I think you definitely have the explanation correct. Unfortunately, when I manually test this query against my database it reverts to the slower method.

– Zev
Mar 26 at 13:27













I have changed the answer with another idea.

– Laurenz Albe
Mar 26 at 13:37





I have changed the answer with another idea.

– Laurenz Albe
Mar 26 at 13:37













Thanks, that works! In the ORM, it was ordering = [F('item_description__product_code_id') + 0]

– Zev
Mar 26 at 13:59






Thanks, that works! In the ORM, it was ordering = [F('item_description__product_code_id') + 0]

– Zev
Mar 26 at 13:59














Following your answer is still giving me the best performance but some db tuning for SSD has helped as well amplitude.engineering/…

– Zev
Mar 27 at 18:50





Following your answer is still giving me the best performance but some db tuning for SSD has helped as well amplitude.engineering/…

– Zev
Mar 27 at 18:50













Sure. Telling your database about your hardware is a good idea. I would also increase effective_io_concurrency.

– Laurenz Albe
Mar 27 at 18:53





Sure. Telling your database about your hardware is a good idea. I would also increase effective_io_concurrency.

– Laurenz Albe
Mar 27 at 18:53








Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.







Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.



















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55344598%2fslow-postgres-query-using-limit%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Kamusi Yaliyomo Aina za kamusi | Muundo wa kamusi | Faida za kamusi | Dhima ya picha katika kamusi | Marejeo | Tazama pia | Viungo vya nje | UrambazajiKuhusu kamusiGo-SwahiliWiki-KamusiKamusi ya Kiswahili na Kiingerezakuihariri na kuongeza habari

Swift 4 - func physicsWorld not invoked on collision? The Next CEO of Stack OverflowHow to call Objective-C code from Swift#ifdef replacement in the Swift language@selector() in Swift?#pragma mark in Swift?Swift for loop: for index, element in array?dispatch_after - GCD in Swift?Swift Beta performance: sorting arraysSplit a String into an array in Swift?The use of Swift 3 @objc inference in Swift 4 mode is deprecated?How to optimize UITableViewCell, because my UITableView lags

Access current req object everywhere in Node.js ExpressWhy are global variables considered bad practice? (node.js)Using req & res across functionsHow do I get the path to the current script with Node.js?What is Node.js' Connect, Express and “middleware”?Node.js w/ express error handling in callbackHow to access the GET parameters after “?” in Express?Modify Node.js req object parametersAccess “app” variable inside of ExpressJS/ConnectJS middleware?Node.js Express app - request objectAngular Http Module considered middleware?Session variables in ExpressJSAdd properties to the req object in expressjs with Typescript