Styjun

I query mongodb via C++. One particular query is oddly returning no results via C++ but works fine at the mongo shell (using the same credentials). I'm trying to understand why they are different and how to make the C++ query work.

This is the code in C++:

bsoncxx::builder::basic::document match_records;
match_records.append(kvp("$and", [&group_id](sub_array sub_and) 
 LOGGABLE;
 sub_and.append([&group_id](sub_document and_sub_doc_1) 
 and_sub_doc_1.append(kvp("$or", [&group_id](sub_array sub_or) 
 sub_or.append([&group_id](sub_document fg_id) 
 fg_id.append(kvp("focus_group_id", group_id.GroupId()));
 );
 sub_or.append([&group_id](sub_document rg_id) 
 rg_id.append(kvp("reading_group_id", group_id.GroupId()));
 );
 ));
 );
 if (something true) 
 sub_and.append([&group_id](sub_document and_sub_doc_2) 
 and_sub_doc_2.append(kvp("$or", [&group_id](sub_array sub_or) 
 sub_or.append([&group_id](sub_document or_sub_stream) 
 or_sub_stream.append(
 kvp("event_category", "focus_group"));
 or_sub_stream.append(
 kvp("event_category", [](sub_document subdoc) 
 subdoc.append(kvp("$exists", 0));
 ));
 );
 ));
 );
 else ... 
 sub_and.append([&group_id](sub_document and_sub_doc_4) 
 and_sub_doc_4.append(kvp("utm", [](sub_document subdoc) 
 subdoc.append(kvp("$exists", 1));
 ));
 );
));
LOG_INFO << bsoncxx::to_json(match_records);

That last log line emits this (which I've reformatted for readability):

 "$and": [
 
 "$or": [
 
 "focus_group_id": 465
 ,
 
 "reading_group_id": 465
 
 ]
 ,
 
 "$or": [
 
 "event_category": "focus_group",
 "event_category": 
 "$exists": 0
 
 
 ]
 ,
 
 "utm": 
 "$exists": 1
 
 
 ]

I execute that query thus:

bsoncxx::builder::basic::document project_fields;
project_fields.append(kvp("_id", 1), kvp("focus_group_id", 1),
 kvp("reading_group_id", 1), kvp("user_id", 1),
 kvp("device_id", 1), kvp("event", 1),
 kvp("event_category", 1), kvp("server_time", 1),
 kvp("details", 1), kvp("utm", 1));
mongocxx::options::find opts;
opts.projection(project_fields.view());
mongocxx::collection candy_events = mongo_db["candy_events"];
mongocxx::cursor cursor = candy_events.find(match_records.view(), opts);
for (const bsoncxx::document::view& doc : cursor) {
 ...

and see that the for loop body is never executed (the query returns no results).

Watching the mongod logs, I see that the query is received with no error flagged:

2019-03-25T08:13:27.864Z I NETWORK [conn6] received client metadata from x,y,z.w:49840 conn6: driver: name: "mongoc / mongocxx", version: "1.13.1-dev / 3.4.0" , os: type: "Linux", name: "Ubuntu", version: "16.04", architecture: "x86_64" , platform: "cfg=0x215680e9 posix=200809 stdc=201112 CC=GCC 5.4.0 20160609 CFLAGS="" LDFLAGS=""" 
2019-03-25T08:13:27.888Z I ACCESS [conn6] Successfully authenticated as principal analytics-staging on example_staging
2019-03-25T08:13:32.104Z I COMMAND [conn6] command example_staging.candy_events command: find find: "candy_events", filter: $and: [ $or: [ focus_group_id: 465 , reading_group_id: 465 ] , $or: [ event_category: "focus_group", event_category: $exists: 0 ] , utm: $exists: 1 ] , projection: _id: 1, focus_group_id: 1, reading_group_id: 1, user_id: 1, device_id: 1, event: 1, event_category: 1, server_time: 1, details: 1, utm: 1 , $db: "example_staging", $readPreference: mode: "primaryPreferred" , lsid: id: UUID("ed852824-f336-40a6-a5c4-54b36439e03b") planSummary: COLLSCAN keysExamined:0 docsExamined:899850 cursorExhausted:1 numYields:7030 nreturned:0 reslen:104 locks: Global: acquireCount: r: 14062 , Database: acquireCount: r: 7031 , Collection: acquireCount: r: 7031 protocol:op_msg 1043ms
2019-03-25T08:13:32.109Z I NETWORK [conn6] end connection x.y.z.w:49840 (1 connection now open)

If I simply copy/paste the query into a db.candy_events.find(), I do get results back and the log shows this:

2019-03-25T08:19:50.566Z I NETWORK [listener] connection accepted from x.y.z.w:40570 #11 (2 connections now open)
2019-03-25T08:19:50.569Z I NETWORK [conn11] received client metadata from x.y.z.w:40570 conn11: application: name: "MongoDB Shell" , driver: name: "MongoDB Internal Client", version: "3.6.5" , os: type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" 
2019-03-25T08:19:50.585Z I ACCESS [conn11] Successfully authenticated as principal analytics-staging on example_staging
2019-03-25T08:19:50.586Z I ACCESS [conn11] Unauthorized: not authorized on admin to execute command getLog: "startupWarnings", $db: "admin" 
2019-03-25T08:19:50.588Z I ACCESS [conn11] Unauthorized: not authorized on admin to execute command replSetGetStatus: 1.0, forShell: 1.0, $db: "admin" 
2019-03-25T08:20:07.031Z I COMMAND [conn11] command example_staging.candy_events appName: "MongoDB Shell" command: find find: "candy_events", filter: $and: [ $or: [ focus_group_id: 465.0 , reading_group_id: 465.0 ] , $or: [ event_category: $exists: 0.0 ] , utm: $exists: 1.0 ] , $db: "example_staging" planSummary: COLLSCAN cursorid:116556503976 keysExamined:0 docsExamined:884119 numYields:6907 nreturned:101 reslen:77090 locks: Global: acquireCount: r: 13816 , Database: acquireCount: r: 6908 , Collection: acquireCount: r: 6908 protocol:op_msg 628ms
2019-03-25T08:20:30.732Z I NETWORK [conn11] end connection x.y.z.w:40570 (1 connection now open)

This is mongod 3.6.5. The difference in query time appears to be cache warming. I suspect the unauthorized admin execution warning is something the shell does at startup, I'm authenticating against example_staging.

Any suggestions what I might be doing wrong or how to go about making this work?

Update / solution

Thanks to @neil-lunn's suggestion to use the profiler, I found that a small mistake in my C++ invocation. That middle clause should have been this:

 if (something true) 
 sub_and.append([&group_id](sub_document and_sub_doc_2) 
 and_sub_doc_2.append(kvp("$or", [&group_id](sub_array sub_or) 
 sub_or.append([&group_id](sub_document is_fg) 
 is_fg.append(kvp("event_category", "focus_group"));
 );
 sub_or.append([&group_id](sub_document is_absent) 
 is_absent.append(
 kvp("event_category", [](sub_document subdoc) 
 subdoc.append(kvp("$exists", 0));
 ));
 );
 ));
 );
 else ...