Remove runtime configurability of core system components

[janl]

Overview

In the 1.x era of CouchDB, many parts of the core systems were managed via the config system. This is mostly due to in the early days, no good standard patterns for what Erlang apps looked like were obvious. This has changed now.

In addition, being able to change core parts of the database, including what code modules to load when and where, and which OS binaries to run when and where, opened us up to a set of security vulnerabilities, that we want to close once and for all with this PR by no longer allowing runtime configuration of core system parts.

Specifically:

• daemons
• [httpd] default_handler
  -httpd_global_handlers
  -httpd_db_handlers
  -httpd_design_handlers
• vhost_global_handlers
• redirect_vhost_handler
• os_daemons
• query_servers
• native_query_servers

This patch retains the ability to configure an existing CouchDB installation to, say, add a third party query server, but it’ll require console access to the server and restarting CouchDB from said console.

---

Details

CouchDB ships with two default query_servers (javascript and coffeescript)
as well as one default native_query_server (query aka mango). These used
to be configured in default.ini in these sections:

[query_servers]
javascript = {{prefix}}/bin/couchjs {{prefix}}/share/server/main.js
coffeescript = {{prefix}}/bin/couchjs {{prefix}}/share/server/main-coffee.js

; enable mango query engine
[native_query_servers]
query = {mango_native_proc, start_link, []}
; erlang query server
; erlang = {couch_native_process, start_link, []}

This allowed end-users post-install and even runtime-changes to which
query servers are enabled and where their binaries live.

This patch changes things, so only a post-install, but not at-runtime
changes are possible from now on.

This still allows people to configure their CouchDB to run a third-
party query server like the somewhat popular Python query server,
but it changes the way the setup is done.

Query Servers

The javascript and coffeescript query servers continue to be enabled
by default. Setup differences have been moved from default.ini to
the couchdb and couchdb.cmd start scripts respectively.

Additional query servers can now be configured using environment
variables:

export COUCHDB_QUERY_SERVER_PYTHON="/path/to/python/query/server.py with args"
couchdb

Where the last segment in the environment variable matches the usual
lowercase(!) query language in the design doc language field.

Multiple query servers can be configured by using more environment
variables.

Native Query Servers

The mango query server continues to be enabled by default. The erlang
query server continues to be disabled by default. This patch adds
a [native_query_servers] enable_erlang_query_server = BOOL setting
(defaults to "false") to enable the erlang query server.

If the legacy configuration for enabling the query server is detected,
that is counted as a true setting as well, so existing configurations
continue to work just fine.

Windows

Since the setting of the ./configure time PREFIX happens during
make release, I had to adapt the couchdb and couchdb.cmd scripts
to have the correct env vars set and the PREFIX replaced there.

I did this to the best of my abilities and research, but this needs
review from the Windows team (Hi Joan! :).

OS Daemons

OS Daemons have been deleted completely as they were deprecated in 2.2.0

Testing recommendations

make check as well as following the instructions above for configuring third party query servers and/or os_daemons as well as enabling the erlang query server.

Checklist

• Code is written and works correctly;
• Changes are covered by tests;
• Documentation reflects the changes;

[janl]

Depends on: apache/couchdb-config#20
Docs PR: apache/couchdb-documentation#328

[janl]

I’m working through the test fails, but this is worth reviewing already.

[wohali]

So far, the couchdb.cmd file looks good, but I haven't run a full test yet. Waiting until the test suite is cleaned up.

[janl]

Latest push also removes couch_httpd_proxy and os_daemons completely

[wohali]

Question: on the COUCHDB_QUERY_SERVER_lang syntax, what happens when lang is erlang? Does this conflict with [native_query_servers] enable_erlang_query_server = true ?

Or query for Mango - same question?

[janl]

@wohali good catch, if someone were to do that, it’d override the internal ones. I think that’s still okay, if someone wanted to write a mango engine in, say, Python. We could forbid the configuring of those two specifically, but it’d be introducing quite fiddly code for not much benefit IMHO.

[janl]

This is good to review now

[rnewson]

I'm +1 on removing runtime configurability but would prefer that the configuration is expressed in the .app files of the respective erlang applications. This will allow package maintainers (etc) to more easily tweak at build time if necessary without introducing the runtime malleability that concerns us.

[janl]

@rnewson I’m happy to add query server defaults to the .app file for your use-case. I also want to retain the possibility for folks who run binary builds to add their third party query servers via env vars.

[rnewson]

{application,appname, [
 {description, "blah"},
 {vsn, git},
 {registered,[]},
 {applications,[kernel,stdlib,config,couch_stats]},
 {mod,{appname_app,[]}},
 {env, [
     {httpd_global_handlers, [
         {_all_dbs, couch_httpd_misc_handlers, handle_all_dbs_req},
         ...},
     ...
   ]}
]}. 

[rnewson]

also noting that changing couchjs stack size is an important ability.

[janl]

also noting that changing couchjs stack size is an important ability.

def agree! we could introduce [query_server_config] js_stack_size (and other options) param, and have it apply to new instances spawned after a config change.

[iilyak]

Did you consider hardcoding it?:

VHostGlobals = ["_utils","_uuids","_session","_users"]

[janl]

good call, might have to keep it dynamic to allow couch.app configurability as per
#1602 (comment) but good call

[wohali]

+1 on the env var approach vs. app files - this makes modification in e.g. Docker containers very straightforward, as well as systemd unit files / init files. "Why not both" is totally fine for me :)

[davisp]

+1 to everything in general. I don't have a strong opinion either way on the environment variable vs. .app config approaches.

[davisp]

It seems a bit silly to keep these as strings just to convert all of the terms in the following list comprehension. I'd at least like to remove that bit of silly for this supervisor. I'd also suggest that we just write out the child specs directly in the SecondarySupervisors list above given that's the more traditional way of doing things (or at least using a helper that gets mapped over the name/mfa pairs.

[janl]

Latest push removes the redundant string parsing. I’d accept commits that turn the list comprehension into a helper, or one that writes it all out.

[wohali]

@janl How is httpsd enabled in this new world? Previously that would have been:

[daemons]
httpsd = {chttpd, start_link, [https]}

[janl]

I'm +1 on removing runtime configurability but would prefer that the configuration is expressed in the .app files of the respective erlang applications. This will allow package maintainers (etc) to more easily tweak at build time if necessary without introducing the runtime malleability that concerns us.

@rnewson latest push defines all http handlers in couch.app.

[janl]

also noting that changing couchjs stack size is an important ability.

Since query servers can still be setup using env vars, the solution to this already exists:

export COUCHDB_QUERY_SERVER_JAVASCRIPT="/path/to/couchjs /path/to/main.js -S STACK SIZE"
couchdb

[janl]

@janl How is httpsd enabled in this new world? Previously that would have been:

@wohali good one, I’ve made this analogous to the native query server:

1. if the legacy config is found, it is treated as option 2. set to true
2. now users can do [ssl] enable = BOOL

[janl]

Pending any test issues and/or review notes, this is good to go now. Thanks everyone for the input!

[rnewson]

very neat, and long overdue.