PepeNOTE: This is my older article moved from medium.com
Query Targeting: Scanned Objects / Returned has gone above 1000
Performance issues can be sometimes a pain in the a**, but I would like to show you how to improve MongoDB query performance on the practical example. I hope it will help you to understand query optimization by creating the proper index.
Prepare the testing environment
I am going to use MongoDB Atlas (simple and fast cluster deployment) and this dataset from kaggle.com. It doesn't matter if you will use Atlas or a self-managed MongoDB cluster. Mostly I will use the CLI tool mongoimport, mongo-shell, or MongoDB Compass. All necessary tools for this exercise, you can find on the MongoDB download page. Mongoimport binary is part of the database tools package. I am not going to describe how to install and use those tools in detail, because there is very good documentation for that.
I have created a dedicated M10 cluster, database user called user, and allow network communication from my IP or 0.0.0.0/0 - not so secure, but it is not a production environment, right? :-)
OK! Now when we have our cluster, let's import the huge dataset. We have two options on how to do it. GUI and CLI. I will describe both.
GUI - MongoDB Compass
This is pretty straightforward. Compass is a very intuitive tool. Firstly we have to create an empty database and collection and then we are able to add data. We will select our dataset file, mark CSV as the input file type and that's it!
CLI - mongoimport
If you are a command line lover, then this is the right option for you. Use mongoimport binary with those parameters:
uri = connection string in quotes
authenticationDatabase = database where your user was created, most common database is admin
db = database name for your import
collection = collection name for your import
drop = delete collection if it is already existing in the cluster, before import starts
type = type of your import file
headerline = the first line will be used as document schema/field names
file = import file
$ mongoimport --uri "mongodb+srv://user:<your-secure-password>@cluster1.xy66x.mongodb.net" --authenticationDatabase admin --db github --collection pull_requests --drop --type=csv --headerline --file=ghtorrent-2019–05–20.csv
Check the collection
We have our github.pull_requests collection in a test environment. Now we can look around with mongoshell for a bit.
$ mongosh "mongodb+srv://user:<your-secure-password>@cluster1.xy66x.mongodb.net"
> show dbs
admin 256.00 KiB
config 268.00 KiB
github 8.51 GiB
local 777.77 MiB
> use github
switched to db github
> show collections
pull_requests
> db.pull_requests.findOne()
{
_id: ObjectId("632080afacad4db4ac0a54f2"),
actor_login: 'EleisonC',
actor_id: 37538393,
comment_id: 197673683,
comment: 'That has been rectified.',
repo: 'openmrs-ocl-client',
author_login: 'Karuhanga',
author_id: 32985500,
pr_id: 41412787,
c_id: 1051024412,
commit_date: ISODate("2019-05-20T06:24:09.000Z")
}
> db.pull_requests.find().count()
81994070
> db.pull_requests.getIndexes()
[ { v: 2, key: { _id: 1 }, name: '_id_' } ]
> db.pull_requests.stats()
{
ns: 'github.pull_requests',
size: Long("26118333454"),
count: 81994070,
avgObjSize: 318,
storageSize: Long("15080230912"),
freeStorageSize: 40480768,
capped: false,
wiredTiger: {
metadata: { formatVersion: 1 },
creationString: 'access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,import=(enabled=false,file_metadata=,repair=false),internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_image_max=0,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,readonly=false,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),type=file,value_format=u,verbose=[],write_timestamp_usage=none',
type: 'file',
uri: 'statistics:table:collection-2-4287955632398155746',
LSM: {
'bloom filter false positives': 0,
'bloom filter hits': 0,
'bloom filter misses': 0,
'bloom filter pages evicted from cache': 0,
'bloom filter pages read into cache': 0,
'bloom filters in the LSM tree': 0,
'chunks in the LSM tree': 0,
'highest merge generation in the LSM tree': 0,
'queries that could have benefited from a Bloom filter that did not exist': 0,
'sleep for LSM checkpoint throttle': 0,
'sleep for LSM merge throttle': 0,
'total size of bloom filters': 0
},
'block-manager': {
'allocations requiring file extension': 504864,
'blocks allocated': 563512,
'blocks freed': 52250,
'checkpoint size': Long("15039696896"),
'file allocation unit size': 4096,
'file bytes available for reuse': 40480768,
'file magic number': 120897,
'file major version number': 1,
'file size in bytes': Long("15080230912"),
'minor version number': 0
},
btree: {
'btree checkpoint generation': 1550,
'btree clean tree checkpoint expiration time': Long("9223372036854775807"),
'btree compact pages reviewed': 0,
'btree compact pages rewritten': 0,
'btree compact pages skipped': 0,
'btree skipped by compaction as process would not reduce size': 0,
'column-store fixed-size leaf pages': 0,
'column-store internal pages': 0,
'column-store variable-size RLE encoded values': 0,
'column-store variable-size deleted values': 0,
'column-store variable-size leaf pages': 0,
'fixed-record size': 0,
'maximum internal page size': 4096,
'maximum leaf page key size': 2867,
'maximum leaf page size': 32768,
'maximum leaf page value size': 67108864,
'maximum tree depth': 4,
'number of key/value pairs': 0,
'overflow pages': 0,
'row-store empty values': 0,
'row-store internal pages': 0,
'row-store leaf pages': 0
},
cache: {
'bytes currently in the cache': 282058257,
'bytes dirty in the cache cumulative': 1895528663,
'bytes read into cache': Long("359307024373"),
'bytes written from cache': Long("29670018281"),
'checkpoint blocked page eviction': 5,
'checkpoint of history store file blocked non-history store page eviction': 0,
'data source pages selected for eviction unable to be evicted': 11185,
'eviction gave up due to detecting an out of order on disk value behind the last update on the chain': 0,
'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update': 0,
'eviction gave up due to detecting an out of order tombstone ahead of the selected on disk update after validating the update chain': 0,
'eviction gave up due to detecting out of order timestamps on the update chain after the selected on disk update':
0,
'eviction walk passes of a file': 91631,
'eviction walk target pages histogram - 0-9': 9364,
'eviction walk target pages histogram - 10-31': 12894,
'eviction walk target pages histogram - 128 and higher': 0,
'eviction walk target pages histogram - 32-63': 16845,
'eviction walk target pages histogram - 64-128': 52528,
'eviction walk target pages reduced due to history store cache pressure': 0,
'eviction walks abandoned': 6316,
'eviction walks gave up because they restarted their walk twice': 20244,
'eviction walks gave up because they saw too many pages and found no candidates': 5531,
'eviction walks gave up because they saw too many pages and found too few candidates': 1050,
'eviction walks reached end of tree': 54382,
'eviction walks restarted': 0,
'eviction walks started from root of tree': 33145,
'eviction walks started from saved location in tree': 58486,
'hazard pointer blocked page eviction': 1085,
'history store table insert calls': 0,
'history store table insert calls that returned restart': 0,
'history store table out-of-order resolved updates that lose their durable timestamp': 0,
'history store table out-of-order updates that were fixed up by reinserting with the fixed timestamp': 0,
'history store table reads': 0,
'history store table reads missed': 0,
'history store table reads requiring squashed modifies': 0,
'history store table truncation by rollback to stable to remove an unstable update': 0,
'history store table truncation by rollback to stable to remove an update': 0,
'history store table truncation to remove an update': 0,
'history store table truncation to remove range of updates due to key being removed from the data page during reconciliation': 0,
'history store table truncation to remove range of updates due to out-of-order timestamp update on data page': 0,
'history store table writes requiring squashed modifies': 0,
'in-memory page passed criteria to be split': 977,
'in-memory page splits': 489,
'internal pages evicted': 36253,
'internal pages split during eviction': 50,
'leaf pages split during eviction': 6354,
'modified pages evicted': 11379,
'overflow pages read into cache': 0,
'page split during eviction deepened the tree': 1,
'page written requiring history store records': 0,
'pages read into cache': 6486249,
'pages read into cache after truncate': 1,
'pages read into cache after truncate in prepare state': 0,
'pages requested from the cache': 181005752,
'pages seen by eviction walk': 47694902,
'pages written from cache': 562692,
'pages written requiring in-memory restoration': 6414,
'the number of times full update inserted to history store': 0,
'the number of times reverse modify inserted to history store': 0,
'tracked dirty bytes in the cache': 0,
'unmodified pages evicted': 6983465
},
cache_walk: {
'Average difference between current eviction generation when the page was last considered': 0,
'Average on-disk page image size seen': 0,
'Average time in cache for pages that have been visited by the eviction server': 0,
'Average time in cache for pages that have not been visited by the eviction server': 0,
'Clean pages currently in cache': 0,
'Current eviction generation': 0,
'Dirty pages currently in cache': 0,
'Entries in the root page': 0,
'Internal pages currently in cache': 0,
'Leaf pages currently in cache': 0,
'Maximum difference between current eviction generation when the page was last considered': 0,
'Maximum page size seen': 0,
'Minimum on-disk page image size seen': 0,
'Number of pages never visited by eviction server': 0,
'On-disk page image sizes smaller than a single allocation unit': 0,
'Pages created in memory and never written': 0,
'Pages currently queued for eviction': 0,
'Pages that could not be queued for eviction': 0,
'Refs skipped during cache traversal': 0,
'Size of the root page': 0,
'Total number of pages currently in cache': 0
},
'checkpoint-cleanup': {
'pages added for eviction': 0,
'pages removed': 0,
'pages skipped during tree walk': 2853123,
'pages visited': 3876644
},
compression: {
'compressed page maximum internal page size prior to compression': 4096,
'compressed page maximum leaf page size prior to compression ': 127796,
'compressed pages read': 6452136,
'compressed pages written': 528035,
'number of blocks with compress ratio greater than 64': 0,
'number of blocks with compress ratio smaller than 16': 0,
'number of blocks with compress ratio smaller than 2': 6447973,
'number of blocks with compress ratio smaller than 32': 0,
'number of blocks with compress ratio smaller than 4': 4163,
'number of blocks with compress ratio smaller than 64': 0,
'number of blocks with compress ratio smaller than 8': 0,
'page written failed to compress': 0,
'page written was too small to compress': 34657
},
cursor: {
'Total number of entries skipped by cursor next calls': 0,
'Total number of entries skipped by cursor prev calls': 0,
'Total number of entries skipped to position the history store cursor': 0,
'Total number of times a search near has exited due to prefix config': 0,
'bulk loaded cursor insert calls': 0,
'cache cursors reuse count': 246011,
'close calls that result in cache': 246014,
'create calls': 32,
'cursor next calls that skip due to a globally visible history store tombstone': 0,
'cursor next calls that skip greater than or equal to 100 entries': 0,
'cursor next calls that skip less than 100 entries': 105,
'cursor prev calls that skip due to a globally visible history store tombstone': 0,
'cursor prev calls that skip greater than or equal to 100 entries': 0,
'cursor prev calls that skip less than 100 entries': 1,
'insert calls': 81994070,
'insert key and value bytes': Long("26511435968"),
modify: 0,
'modify key and value bytes affected': 0,
'modify value bytes modified': 0,
'next calls': 105,
'open cursor count': 4,
'operation restarted': 0,
'prev calls': 1,
'remove calls': 0,
'remove key bytes removed': 0,
'reserve calls': 0,
'reset calls': 1762637,
'search calls': 1226093920,
'search history store calls': 0,
'search near calls': 1,
'truncate calls': 0,
'update calls': 0,
'update key and value bytes': 0,
'update value size change': 0
},
reconciliation: {
'approximate byte size of timestamps in pages written': 1355280480,
'approximate byte size of transaction IDs in pages written': 677640240,
'dictionary matches': 0,
'fast-path pages deleted': 0,
'internal page key bytes discarded using suffix compression': 535709,
'internal page multi-block writes': 811,
'leaf page key bytes discarded using prefix compression': 0,
'leaf page multi-block writes': 6732,
'leaf-page overflow keys': 0,
'maximum blocks required for a page': 31,
'overflow values written': 0,
'page checksum matches': 0,
'page reconciliation calls': 12985,
'page reconciliation calls for eviction': 9937,
'pages deleted': 0,
'pages written including an aggregated newest start durable timestamp ': 33861,
'pages written including an aggregated newest stop durable timestamp ': 0,
'pages written including an aggregated newest stop timestamp ': 0,
'pages written including an aggregated newest stop transaction ID': 0,
'pages written including an aggregated newest transaction ID ': 33861,
'pages written including an aggregated oldest start timestamp ': 33861,
'pages written including an aggregated prepare': 0,
'pages written including at least one prepare': 0,
'pages written including at least one start durable timestamp': 527834,
'pages written including at least one start timestamp': 527834,
'pages written including at least one start transaction ID': 527834,
'pages written including at least one stop durable timestamp': 0,
'pages written including at least one stop timestamp': 0,
'pages written including at least one stop transaction ID': 0,
'records written including a prepare': 0,
'records written including a start durable timestamp': 84705030,
'records written including a start timestamp': 84705030,
'records written including a start transaction ID': 84705030,
'records written including a stop durable timestamp': 0,
'records written including a stop timestamp': 0,
'records written including a stop transaction ID': 0
},
session: {
'object compaction': 0,
'tiered operations dequeued and processed': 0,
'tiered operations scheduled': 0,
'tiered storage local retention time (secs)': 0
},
transaction: {
'race to read prepared update retry': 0,
'rollback to stable history store records with stop timestamps older than newer records': 0,
'rollback to stable inconsistent checkpoint': 0,
'rollback to stable keys removed': 0,
'rollback to stable keys restored': 0,
'rollback to stable restored tombstones from history store': 0,
'rollback to stable restored updates from history store': 0,
'rollback to stable skipping delete rle': 0,
'rollback to stable skipping stable rle': 0,
'rollback to stable sweeping history store keys': 0,
'rollback to stable updates removed from history store': 0,
'transaction checkpoints due to obsolete pages': 0,
'update conflicts': 0
}
},
nindexes: 1,
indexBuilds: [],
totalIndexSize: Long("2582536192"),
totalSize: Long("17662767104"),
indexSizes: { _id_: Long("2582536192") },
scaleFactor: 1,
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1663158374, i: 22 }),
signature: {
hash: Binary(Buffer.from("6d46acf48f3154a0144772ca4187aeed7d480715", "hex"), 0),
keyId: Long("7142807608573820932")
}
},
operationTime: Timestamp({ t: 1663158374, i: 22 })
}
> db.getCollectionInfos()
[
{
name: 'pull_requests',
type: 'collection',
options: {},
info: {
readOnly: false,
uuid: UUID("0081a40b-8a0c-47a0-81f8-10aaabbb0da5")
},
idIndex: { v: 2, key: { _id: 1 }, name: '_id_' }
}
]
Great! Now we have an overview of our collection. It consists of 81994070 documents and has only one index _id. Ideal case for triggering a very slow query :-D
Slow Down!!!
Hmm, let's create a complex query to see what will happened. We are going to list all documents where actor_login is EleisonC, author_id is 32985500, and commit_date older than 2019–05–20 16:56:17. In the end we are going to sort the output by _id, where we are sure that has index already.
> db.pull_requests.find({"$and":[{"actor_login":"EleisonC"},{"author_id": 32985500},{"commit_date":{$lte:ISODate("2019-05-20T16:56:17.000Z")}}]}).sort({_id: 1})
Oooooh seems pretty slow, about 4 minutes to get results. In the meantime, we could check it with currentOp in a different session (needed role dbAdmin).
> db.currentOp( {"$ownOps": true } )
{
inprog: [
{
type: 'op',
host: 'atlas-u3sxt5-shard-00-01.xy66x.mongodb.net:27017',
desc: 'conn13447',
connectionId: 13447,
client: '192.168.1.195:24750',
appName: 'mongosh 1.5.4',
clientMetadata: {
driver: { name: 'nodejs|mongosh', version: '4.8.1' },
os: {
type: 'Linux',
name: 'linux',
architecture: 'x64',
version: '5.10.16.3-microsoft-standard-WSL2'
},
platform: 'Node.js v16.16.0, LE (unified)',
version: '4.8.1|1.5.4',
application: { name: 'mongosh 1.5.4' }
},
active: true,
currentOpTime: '2022-09-14T13:03:38.070+00:00',
effectiveUsers: [ { user: 'user', db: 'admin' } ],
threaded: true,
opid: 6766360,
lsid: {
id: UUID("9cd19c2a-7e03-447f-b131-69c507270142"),
uid: Binary(Buffer.from("78e672a9f1df895a5fbe23c35a0004466ccd167529f1405d95628de7416ff5fa", "hex"), 0)
},
secs_running: Long("0"),
microsecs_running: Long("120"),
op: 'command',
ns: 'admin.$cmd.aggregate',
command: {
currentOp: 1,
'$ownOps': true,
lsid: { id: UUID("9cd19c2a-7e03-447f-b131-69c507270142") },
'$clusterTime': {
clusterTime: Timestamp({ t: 1663160603, i: 1 }),
signature: {
hash: Binary(Buffer.from("7f6f11951f921b79bdf4e8bd2e702cb238b17e00", "hex"), 0),
keyId: Long("7142807608573820932")
}
},
'$db': 'admin'
},
numYields: 0,
locks: {},
waitingForLock: false,
lockStats: {},
waitingForFlowControl: false,
flowControlStats: {}
},
{
type: 'op',
host: 'atlas-u3sxt5-shard-00-01.xy66x.mongodb.net:27017',
desc: 'conn12942',
connectionId: 12942,
client: '192.168.1.195:24389',
appName: 'mongosh 1.5.4',
clientMetadata: {
driver: { name: 'nodejs|mongosh', version: '4.8.1' },
os: {
type: 'Linux',
name: 'linux',
architecture: 'x64',
version: '5.10.16.3-microsoft-standard-WSL2'
},
platform: 'Node.js v16.16.0, LE (unified)',
version: '4.8.1|1.5.4',
application: { name: 'mongosh 1.5.4' }
},
active: true,
currentOpTime: '2022-09-14T13:03:38.070+00:00',
effectiveUsers: [ { user: 'user', db: 'admin' } ],
threaded: true,
opid: 6765691,
lsid: {
id: UUID("3d3c092e-3187-42f2-a0d6-ed30d61dc571"),
uid: Binary(Buffer.from("78e672a9f1df895a5fbe23c35a0004466ccd167529f1405d95628de7416ff5fa", "hex"), 0)
},
secs_running: Long("19"),
microsecs_running: Long("19815843"),
op: 'query',
ns: 'github.pull_requests',
command: {
find: 'pull_requests',
filter: {
'$and': [
{ actor_login: 'EleisonC' },
{ author_id: 32985500 },
{ commit_date: [Object] }
]
},
sort: { _id: 1 },
lsid: { id: UUID("3d3c092e-3187-42f2-a0d6-ed30d61dc571") },
'$clusterTime': {
clusterTime: Timestamp({ t: 1663159643, i: 1 }),
signature: {
hash: Binary(Buffer.from("acde94b5658281a5b9eea21b0ffaae6f0653465f", "hex"), 0),
keyId: Long("7142807608573820932")
}
},
'$db': 'github'
},
planSummary: 'IXSCAN { _id: 1 }',
numYields: 9194,
locks: { FeatureCompatibilityVersion: 'r', Global: 'r' },
waitingForLock: false,
lockStats: {
FeatureCompatibilityVersion: { acquireCount: { r: Long("9195") } },
Global: { acquireCount: { r: Long("9195") } },
Mutex: { acquireCount: { r: Long("1") } }
},
waitingForFlowControl: false,
flowControlStats: {}
}
],
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1663160613, i: 1 }),
signature: {
hash: Binary(Buffer.from("bd234febc72dd8fa57796adbfd7fde21e6aa33ab", "hex"), 0),
keyId: Long("7142807608573820932")
}
},
operationTime: Timestamp({ t: 1663160613, i: 1 })
}
If you are not able to catch the query with db command, check logfiles. There will be all the necessary information you need.
mongodb.log:
{"t":{"$date":"2022-09-14T13:13:33.361+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn12942","msg":"Slow query","attr":{"type":"command","ns":"github.pull_requests","appName":"mongosh 1.5.4","command":{"find":"pull_requests","filter":{"$and":[{"actor_login":"EleisonC"},{"author_id":32985500},{"commit_date":{"$lte":{"$date":"2019-05-20T16:56:17.000Z"}}}]},"sort":{"_id":1},"lsid":{"id":{"$uuid":"3d3c092e-3187-42f2-a0d6-ed30d61dc571"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1663160774,"i":23}},"signature":{"hash":{"$binary":{"base64":"kHRt5Xd0iUjWwslfWxjvuRcIxUA=","subType":"0"}},"keyId":7142807608573820932}},"$db":"github"},"planSummary":"IXSCAN { _id: 1 }","keysExamined":81994070,"docsExamined":81994070,"cursorExhausted":true,"numYields":82007,"nreturned":74,"queryHash":"33DBB573","planCacheKey":"98434520","reslen":21182,"locks":{"FeatureCompatibilityVersion":{"acquireCount":{"r":82008}},"Global":{"acquireCount":{"r":82008}},"Mutex":{"acquireCount":{"r":1}}},"readConcern":{"level":"local","provenance":"implicitDefault"},"storage":{"data":{"bytesRead":31566325989,"timeReadingMicros":54395866},"timeWaitingMicros":{"cache":8538}},"remote":"196.168.1.195:24389","protocol":"op_msg","durationMillis":181936}}
When we are checking the situation of query execution with db.currentOp() a couple of times, we can see that number of seconds is increasing, the plan is using index (IXSCAN) and numYields are increasing as well, which is not good. The same behavior is repeated every time we execute the query. That means the query is reading from disk (all the time) and it is inefficient. In logfile we can see another issue. We are scanning all documents for 74 results. That is a very bad Scanned/Returned ratio. But why? we are using the index, right? Looks like our existing index is not enough for processing such an amount of data and the query needs something more sophisticated.
How to choose the proper index?
Well, there are some rules. Ideally, the index might cover the whole query and be created by the ESR rule. Hmm okay, so how do we put four fields into one index? MongoDB supports a lot of index type and one of them is a compound index. To cover a query we have to use fields: actor_login, author_id, commit_date, _id.
> db.pull_requests.find({"$and":[{"actor_login":"EleisonC"},{"author_id": 32985500},{"commit_date":{$lte:ISODate("2019-05-20T16:56:17.000Z")}}]}).sort({_id: 1})
Fine, should I add all fields in order as they are used in the query? That's not a good solution for that. As I mentioned earlier we have to follow the ESR rule.
the ESR Rule - Equality, Sort, Range
Order of fields in compound index start with fields with exact match {"actor_login":"EleisonC"},{"author_id": 32985500} (equality), then follows fields used for sorting operation sort({_id: 1}) (sort) and last one is field with not exact match, but when something is less than $lt , greater than $gt, not equal $ne, etc. (range)
>db.pull_requests.createIndex( { "actor_login" : 1, "author_id" : 1, "_id" : 1, "commit_date" : 1} )
This index is our WINNER!!!
Here you can see the difference between the index actor_login_1_author_id_1id_1_commit_date_1 and the index actor_login_1_author_id_1_commit_date_1id_1.
> db.pull_requests.explain("allPlansExecution").find({"$and":[{"actor_login":"EleisonC"},{"author_id": 32985500},{"commit_date":{$lte:ISODate("2019-05-20T16:56:17.000Z")}}]}).sort({_id: 1})
{
explainVersion: '1',
queryPlanner: {
namespace: 'github.pull_requests',
indexFilterSet: false,
parsedQuery: {
'$and': [
{ actor_login: { '$eq': 'EleisonC' } },
{ author_id: { '$eq': 32985500 } },
{
commit_date: { '$lte': ISODate("2019-05-20T16:56:17.000Z") }
}
]
},
maxIndexedOrSolutionsReached: false,
maxIndexedAndSolutionsReached: false,
maxScansToExplodeReached: false,
winningPlan: {
stage: 'FETCH',
inputStage: {
stage: 'IXSCAN',
keyPattern: { actor_login: 1, author_id: 1, _id: 1, commit_date: 1 },
indexName: 'actor_login_1_author_id_1__id_1_commit_date_1',
isMultiKey: false,
multiKeyPaths: { actor_login: [], author_id: [], _id: [], commit_date: [] },
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: 2,
direction: 'forward',
indexBounds: {
actor_login: [ '["EleisonC", "EleisonC"]' ],
author_id: [ '[32985500, 32985500]' ],
_id: [ '[MinKey, MaxKey]' ],
commit_date: [
'[new Date(-9223372036854775808), new Date(1558371377000)]'
]
}
}
},
rejectedPlans: [
{
stage: 'FETCH',
inputStage: {
stage: 'SORT',
sortPattern: { _id: 1 },
memLimit: 104857600,
type: 'default',
inputStage: {
stage: 'IXSCAN',
keyPattern: { actor_login: 1, author_id: 1, commit_date: 1, _id: 1 },
indexName: 'actor_login_1_author_id_1_commit_date_1__id_1',
isMultiKey: false,
multiKeyPaths: {
actor_login: [],
author_id: [],
commit_date: [],
_id: []
},
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: 2,
direction: 'forward',
indexBounds: {
actor_login: [ '["EleisonC", "EleisonC"]' ],
author_id: [ '[32985500, 32985500]' ],
commit_date: [
'[new Date(-9223372036854775808), new Date(1558371377000)]'
],
_id: [ '[MinKey, MaxKey]' ]
}
}
}
}
]
},
executionStats: {
executionSuccess: true,
nReturned: 74,
executionTimeMillis: 1,
totalKeysExamined: 75,
totalDocsExamined: 74,
executionStages: {
stage: 'FETCH',
nReturned: 74,
executionTimeMillisEstimate: 0,
works: 76,
advanced: 74,
needTime: 0,
needYield: 0,
saveState: 0,
restoreState: 0,
isEOF: 1,
docsExamined: 74,
alreadyHasObj: 0,
inputStage: {
stage: 'IXSCAN',
nReturned: 74,
executionTimeMillisEstimate: 0,
works: 75,
advanced: 74,
needTime: 0,
needYield: 0,
saveState: 0,
restoreState: 0,
isEOF: 1,
keyPattern: { actor_login: 1, author_id: 1, _id: 1, commit_date: 1 },
indexName: 'actor_login_1_author_id_1__id_1_commit_date_1',
isMultiKey: false,
multiKeyPaths: { actor_login: [], author_id: [], _id: [], commit_date: [] },
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: 2,
direction: 'forward',
indexBounds: {
actor_login: [ '["EleisonC", "EleisonC"]' ],
author_id: [ '[32985500, 32985500]' ],
_id: [ '[MinKey, MaxKey]' ],
commit_date: [
'[new Date(-9223372036854775808), new Date(1558371377000)]'
]
},
keysExamined: 75,
seeks: 1,
dupsTested: 0,
dupsDropped: 0
}
},
allPlansExecution: [
{
nReturned: 74,
executionTimeMillisEstimate: 0,
totalKeysExamined: 75,
totalDocsExamined: 74,
executionStages: {
stage: 'FETCH',
nReturned: 74,
executionTimeMillisEstimate: 0,
works: 75,
advanced: 74,
needTime: 0,
needYield: 0,
saveState: 0,
restoreState: 0,
isEOF: 1,
docsExamined: 74,
alreadyHasObj: 0,
inputStage: {
stage: 'IXSCAN',
nReturned: 74,
executionTimeMillisEstimate: 0,
works: 75,
advanced: 74,
needTime: 0,
needYield: 0,
saveState: 0,
restoreState: 0,
isEOF: 1,
keyPattern: { actor_login: 1, author_id: 1, _id: 1, commit_date: 1 },
indexName: 'actor_login_1_author_id_1__id_1_commit_date_1',
isMultiKey: false,
multiKeyPaths: {
actor_login: [],
author_id: [],
_id: [],
commit_date: []
},
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: 2,
direction: 'forward',
indexBounds: {
actor_login: [ '["EleisonC", "EleisonC"]' ],
author_id: [ '[32985500, 32985500]' ],
_id: [ '[MinKey, MaxKey]' ],
commit_date: [
'[new Date(-9223372036854775808), new Date(1558371377000)]'
]
},
keysExamined: 75,
seeks: 1,
dupsTested: 0,
dupsDropped: 0
}
}
},
{
nReturned: 0,
executionTimeMillisEstimate: 0,
totalKeysExamined: 74,
totalDocsExamined: 0,
executionStages: {
stage: 'FETCH',
nReturned: 0,
executionTimeMillisEstimate: 0,
works: 75,
advanced: 0,
needTime: 75,
needYield: 0,
saveState: 0,
restoreState: 0,
isEOF: 0,
docsExamined: 0,
alreadyHasObj: 0,
inputStage: {
stage: 'SORT',
nReturned: 0,
executionTimeMillisEstimate: 0,
works: 75,
advanced: 0,
needTime: 75,
needYield: 0,
saveState: 0,
restoreState: 0,
isEOF: 0,
sortPattern: { _id: 1 },
memLimit: 104857600,
type: 'default',
totalDataSizeSorted: 6068,
usedDisk: false,
inputStage: {
stage: 'IXSCAN',
nReturned: 74,
executionTimeMillisEstimate: 0,
works: 75,
advanced: 74,
needTime: 0,
needYield: 0,
saveState: 0,
restoreState: 0,
isEOF: 1,
keyPattern: { actor_login: 1, author_id: 1, commit_date: 1, _id: 1 },
indexName: 'actor_login_1_author_id_1_commit_date_1__id_1',
isMultiKey: false,
multiKeyPaths: {
actor_login: [],
author_id: [],
commit_date: [],
_id: []
},
isUnique: false,
isSparse: false,
isPartial: false,
indexVersion: 2,
direction: 'forward',
indexBounds: {
actor_login: [ '["EleisonC", "EleisonC"]' ],
author_id: [ '[32985500, 32985500]' ],
commit_date: [
'[new Date(-9223372036854775808), new Date(1558371377000)]'
],
_id: [ '[MinKey, MaxKey]' ]
},
keysExamined: 74,
seeks: 1,
dupsTested: 0,
dupsDropped: 0
}
}
}
}
]
},
command: {
find: 'pull_requests',
filter: {
'$and': [
{ actor_login: 'EleisonC' },
{ author_id: 32985500 },
{
commit_date: { '$lte': ISODate("2019-05-20T16:56:17.000Z") }
}
]
},
sort: { _id: 1 },
'$db': 'github'
},
serverInfo: {
host: 'atlas-u3sxt5-shard-00-01.xy66x.mongodb.net',
port: 27017,
version: '5.0.12',
gitVersion: '79cfcdd83eb6f64e164a588d0daf9bb873328b45'
},
serverParameters: {
internalQueryFacetBufferSizeBytes: 104857600,
internalQueryFacetMaxOutputDocSizeBytes: 104857600,
internalLookupStageIntermediateDocumentMaxSizeBytes: 104857600,
internalDocumentSourceGroupMaxMemoryBytes: 104857600,
internalQueryMaxBlockingSortMemoryUsageBytes: 104857600,
internalQueryProhibitBlockingMergeOnMongoS: 0,
internalQueryMaxAddToSetBytes: 104857600,
internalDocumentSourceSetWindowFieldsMaxMemoryBytes: 104857600
},
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1663178353, i: 1 }),
signature: {
hash: Binary(Buffer.from("1834928e18dd57e09c13189fd097e110caac7d85", "hex"), 0),
keyId: Long("7142807608573820932")
}
},
operationTime: Timestamp({ t: 1663178353, i: 1 })
}
The compound indexes are good for queries that are executed very often and selected fields are stable.
This article was written as some kind of working note for me, but I hope somebody will find it useful :-)
Sources:
