Autocompaction never triggered in Couch v2.2

[akircher]

Expected Behavior

Autocompaction is not being triggered even though the db and views are above the threshold % specified in the config and there seems to be enough diskspace: ~30% used on a 150GB drive. No error is provided in the log file. Triggering compaction manually with curl works as expected.

My compaction settings are currently but I have tried many variations of these

compaction_daemon | check_interval | 30 |  
compaction_daemon | check_interval | min_file_size | 13000
compactions | _default | [{db_fragmentation, "20%"}, {view_fragmentation, "20%"}]

My installation has 5 databases. /_dbs:

{"db_name":"_dbs","update_seq":"7-g2wAAAABaANkABFjb3VjaGRiQDEyNy4wLjAuMWwAAAACYQBuBAD_____amEHag","sizes":{"file":12479,"external":8365,"active":3004},"purge_seq":0,"other":{"data_size":8365},"doc_del_count":0,"doc_count":7,"disk_size":12479,"disk_format_version":6,"data_size":3004,"compact_running":false,"cluster":{"q":1,"n":1,"w":1,"r":1},"instance_start_time":"0"}

The largest db is transasction: /transaction:
(note that I just ran compaction manually on this and so it no longer needs an autocompaction)

{"db_name":"transaction","update_seq":"913507-g1AAAAFreJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____s5IYGPcZ41GXpAAkk-yhSvf441PqAFIaD1W6owSf0gSQ0nqo0kPX8SjNYwGSDA1ACqh6Pkj5ri0ElS-AKN8PUr4_mqDyAxDl90HKD2wjqPwBRDnY7fsvZgEAVlBo7Q","sizes":{"file":260130402,"external":218239373,"active":259627940},"purge_seq":0,"other":{"data_size":218239373},"doc_del_count":4816,"doc_count":273983,"disk_size":260130402,"disk_format_version":6,"data_size":259627940,"compact_running":false,"cluster":{"q":8,"n":1,"w":1,"r":1},"instance_start_time":"0"}

I have about 40 views each within their own ddoc. Here is one of the 40 views that are not getting compacting /transaction/_design/inventory.value-by-generic/_info

{"name":"inventory.value-by-generic","view_index":{"updates_pending":{"minimum":0,"preferred":0,"total":0},"waiting_commit":false,"waiting_clients":0,"updater_running":false,"update_seq":913507,"sizes":{"file":2205017462,"external":258446339,"active":217542472},"signature":"e31ca84e6148d04f91b6fff3a5f3c329","purge_seq":0,"language":"javascript","disk_size":2205017462,"data_size":258446339,"compact_running":false}}

I have been calculating fragmentation as (2205017462-258446339)/2205017462 = 88%

Possible Solution

I am thinking it is a bug with 2.x Things seemed to be working on 1.7. Just a guess but my view names do contain a . which might be getting confused with the how shards are identified in the compaction syntax.

Context

I have to run compaction manually on all databases and views.

Your Environment

• Version used:
• Browser Name and version:
• Operating System and version (desktop or mobile):
• Link to your project:
  Ubuntu 16.04. Couch 2.2. Node 10

[regner]

This is happening for us as well. We have roughly 400 GB of DB files and this hits us pretty hard as some of the DBs see a lot of changes.

Our DB disk before the upgrade was hovering just below 60%. You can see the daily compaction (we have compaction restricted to off hours) take the disk space down a bit every day as it goes up. then on the night of September 6th we deployed CouchDB 2.2 and compaction just kinda stops happening. Disk space usage grew at a fairly steady pace and our alerts triggered at 85%.

After a bunch of reviewing our configs, testing different configs, we were unable to get auto compaction to work. On the 23rd we ran a script that grabbed every DB, compared the disk size to data size, and then we manually compacted (using the _compact endpoint) the worst 1k DBs. This gave us back about 22% of our disk, but is not a long term solution.

On one of our other DB servers that sees a lot of activity the _global_changes DB grew to about 85 GB in size. We manually triggered the compaction and it compacted down to 1 GB.

We have been playing with a bunch of different values to see if something else works, but here are some of our current config settings:

[compaction_daemon]
check_interval = 3600
min_file_size = 131072 (most our DBs are 500 MB in size)

[compactions]
_default = [{db_fragmentation, "40%"}, {view_fragmentation, "30%"}, {from, "03:00"}, {to, "09:00"}]

We did confirm CouchDB is reading the compactions settings correctly and that the compaction daemon is running.

We do configure CouchDB to store all data on the /srv mount, but there are no symlinks.

If there is any other information you need or that we can provide let us know. :)

[wohali]

@regner Can you confirm that your upgrade was from 2.1.0 to 2.2.0 and that the problems only started happening with 2.2.0?

[regner]

Will check git history of our puppet configs to get you the exact version numbers.

[regner]

We went from 2.1.1 to 2.2.0 so we did skip the 2.1.2 version.

[nickva]

Could it be that we are not reading the design docs correctly:

https://github.com/apache/couchdb/blob/master/src/couch/src/couch_compaction_daemon.erl#L241

Notice we just get the design docs from each shard as we look through all db shard. However design docs won't be written to all the shards of a database. They'll go to whatever shard range their doc id would be hashed to.

Instead we need to get this design docs from the fabric layer if this is a clustered db. Something like:

fabric:design_docs(mem3:dbname(DbName))

[nickva]

The outcome of the bug described above is that only the view shards where the _design doc live (1 out of 8 by default) will be compacted.

Something like this:

ls -l */thedb*/mrview

00000000-1fffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 392
-rw-r--r--  1 staff  135304 Nov 27 21:12 8f3db710871cdc942f4793936e3de269.view

20000000-3fffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 264
-rw-r--r--  1  staff  127112 Nov 27 21:12 8f3db710871cdc942f4793936e3de269.view

40000000-5fffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 264
-rw-r--r--  1 staff  118920 Nov 27 21:12 8f3db710871cdc942f4793936e3de269.view

60000000-7fffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 296
-rw-r--r--  1 staff  86152 Nov 27 21:38 8f3db710871cdc942f4793936e3de269.view

80000000-9fffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 264
-rw-r--r--  1 staff  127112 Nov 27 21:12 8f3db710871cdc942f4793936e3de269.view

a0000000-bfffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 264
-rw-r--r--  1 staff  123016 Nov 27 21:12 8f3db710871cdc942f4793936e3de269.view

c0000000-dfffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 264
-rw-r--r--  1 staff  127112 Nov 27 21:12 8f3db710871cdc942f4793936e3de269.view

e0000000-ffffffff/thedb-e1105412deb04d7d82e827185fca5fdf.1543358834_design/mrview:
total 264
-rw-r--r--  1 staff  123016 Nov 27 21:12 8f3db710871cdc942f4793936e3de269.view

Design document in this case lives in the 60000000-7fffffff range

[nickva]

@akircher @regner

Thank you for your reports and sorry for the delay investigating this. I have theory why this might be happening. Would you be able to inspect your setup to confirm if you see the same pattern. I think this is at least one possible bug in that code. And it would be consistent with the observation that it would have worked in 1.x but not in 2.x

So basically in your data directory pick a data base name and run this command:

ls -l .shards/*/thedb*/mrview

(thedb if your database name in this case). And see if you get the same pattern, where most view shard are not compacted but one is.

[regner]

We just ran a compaction of all our databases with our stop-gap script so not sure if that will change the outcome of that query.

Also I should note that our config only has 1 shard. Specifically:

[cluster]
q=1
n=6

I will try and run that query later this evening or tomorrow though and get you the results.

[nickva]

@regner Hmm, if it is just one shard it's probably a different issue. In that case try to enable debug logging and see if there are any messages from the compaction daemon in the log.

Search for couch_log: in https://github.com/apache/couchdb/blob/master/src/couch/src/couch_compaction_daemon.erl and see what some of the messages might look like. It would be things like Compaction daemon - canceling compaction for databases ... because exceeded the allowed time."

Another thing to try is to remove the time constraints and see if it works. Noticed above your setup had a from and to set.