Update default config settings (Q, max_document_size, etc.)

[kocolosk]

I would like to take the opportunity to revisit a few default configuration settings for 3.0 deployments:

cluster.q (currently 8) - we have had mailing list threads about this is too large for many deployments. The introduction of shard splitting gives us more flexibility to add shards to a database after the fact. I'd suggest setting this to 1 or 2 as a default.

couchdb.max_document_size - the current default of 4GB is quite insane. The view engine already cannot handle anything much larger than ~10MB without a lot of tweaking. Moreover, 4.0 is going to force our hand on making documents smaller. I'd suggest we lower this default to 1MB.

[rnewson]

+1 on lowering max_document_size to 1MB.

default q is more controversial. How well is the shard splitting working? If it's seamless (no interruption to reads or writes) and efficient, then lowering to 2 is ok by me (I'd not be keen on 1). That said, q vanishes entirely in 4.0 and we've lived with this default for a long time. Is it worth changing at this late stage?

[wohali]

There was significant discussion on both of these changes in other tickets and on the mailing list.

I'm -1 until we at least review that feedback and consider the implications.

I believe we got closer to q=2 or 4 on the lists than 1 or 2. Or we could be more intelligent and try and autopick on first run or something, based on # of CPU cores and nodes in the cluster, and th absence of a q= value in the local ini files...spitballing here.

As for the max document size, I think this is actually more controversial, and will break anyone who has larger attachments in CouchDB today who upgrades! That seems to violate the principle of least surprise. See #1253 and #1200 for the latest on this.

/cc @janl

[kocolosk]

I erred on the side of fewer shards because we have splitting but not merging, and because the initial setting of 8 was derived from some of our deployments which I'd wager are larger than average. But I'm happy to go along with whatever consensus (including retaining the current default if that's what folks want).

Attachments aren't included in the computation for couchdb.max_document_size, only the actual JSON fields and values. That said, there are absolutely users in the wild with JSON docs larger than 1 MB, and this change in defaults could certainly surprise them. In 3.x they'll be able to work around it by changing the default on their server, but this is meant as a signal to re-think the data model. My experience has been that those large docs are problematic for a number of reasons; the somewhat fuzzy limit at which the view engine breaks, and the intense write amplification when updating any field of the document are two examples that come to mind.

[wohali]

I'd be fine with q=2 or q=4 as a default. I guess that means we mostly agree on q=2, so +1 on that.

I am -1 on the default attachment size reduction to anything below 10MB. Are we not going to be able to handle 10MB attachments natively in 4.0 even? I'm fine with signalling to users that 350MB attachments are a concern, but 1MB is hardly enough to include a photo or short video from an IoT security camera. 10MB seems like a fair compromise to me.

[kocolosk]

OK. I wasn’t proposing an attachment size limit. I was proposing a limit on the document size, excluding attachment bodies.

[wohali]

@kocolosk Sorry, been task switching a lot, and I clearly misread here.

What happens if a user upgrades in place to CouchDB 3.x and they have documents already >1MB of JSON, but the limit is reduced? What behaviour do they experience?

[kocolosk]

The database will continue to serve those documents, but any subsequent updates to the documents will be rejected unless the new version of the document is < 1 MB

[kocolosk]

Closed via #2228 - we settled on q=2 and max_document_size=8MB for now