Erlang crash dump written on no admin user

[wohali]

Since #2389 when running CouchDB without an admin user (such as make release && rel/couchdb/bin/couchdb), we get an Erlang crash dump written to disk:

[info] 2020-01-06T19:01:44.584000Z couchdb@localhost <0.235.0> --------
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  No Admin Account Found, aborting startup.
  Please configure an admin account in your local.ini file.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[error] 2020-01-06T19:01:45.100000Z couchdb@localhost <0.234.0> -------- CRASH REPORT Process  (<0.234.0>) with 0 neighbors exited with reason: bad return value admin_account_required from couch_app:start(normal, []) at application_master:init/4(line:134) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {application_master,init,['Argument__1','Argument__2',...]}, ancestors: [<0.233.0>], messages: [{'EXIT',<0.235.0>,normal}], links: [<0.233.0>,<0.9.0>], dictionary: [], trap_exit: true, status: running, heap_size: 610, stack_size: 27, reductions: 114
[info] 2020-01-06T19:01:45.100000Z couchdb@localhost <0.9.0> -------- Application couch exited with reason: bad return value admin_account_required from couch_app:start(normal, [])
{"Kernel pid terminated",application_controller,"{application_start_failure,couch,{bad_return,{{couch_app,start,[normal,[]]},admin_account_required}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,couch,{bad_return,{{couch_app,start,[normal,[]]},admin_account_required}}})

Crash dump is being written to: erl_crash.dump...done

We should exit with a non-zero exit code, but not write a crash dump if at all possible.

/cc @janl

[chewbranca]

We can improve the error returned in the stacktrace, but eliminating that stack trace and crash dump has some tradeoffs.

For the returned error, the relevant bit is: bad return value admin_account_required. This is not an acceptable return value out of an application start callback. It needs to be ok or {error, Reason}, as per [1]. It appears that the ok or error tuple response also applies to thrown errors. I'm having trouble finding the documentation around it, but supervisor:start_link handles thrown errors if they're error tuples, which is what is bubbling up by way of [2]. If we switch the throw(admin_account_required) in [2] to be throw({error, admin_account_required}) then we get an improved error message with:

(! 2533)-> ./rel/couchdb/bin/couchdb                                                                                                   
[info] 2020-01-06T21:18:16.696903Z couchdb@127.0.0.1 <0.259.0> -------- Preflight check: Asserting Admin Account
                                                                   
[info] 2020-01-06T21:18:16.696973Z couchdb@127.0.0.1 <0.259.0> -------- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%                                                                          
  No Admin Account Found, aborting startup.                        
  Please configure an admin account in your local.ini file.        
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%                                                                          
                                                                   
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed          
{"Kernel pid terminated",application_controller,"{application_start_failure,couch,{admin_account_required,{couch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,couch,{admin_account_required,{couch_app,start,[normal,[]]}}})
                                                                                                                                       
Crash dump is being written to: erl_crash.dump...done                                                                                  

This still has the kernel shutdown and crash dump, but it's at least a proper application_start_failure rather than a bad_return error, so at the very least this would be a correct fix to the current error. This fix also has the property of placing nicely with "CouchDB as a library," in the sense of embedding CouchDB in another application or even just starting it manually in an Erlang shell. For instance:

(! 2544)-> erl -pa rel/couchdb/lib/*/ebin                          
Erlang/OTP 22 [erts-10.5.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]                                  
                                                                                                                                       
Eshell V10.5.1  (abort with ^G)                                                                                                                                                                                                                                               
1> application:start(couch).                                                                                                                                                                                                                                                  
{error,{not_started,crypto}}                                       
2> application:ensure_all_started(couch).                          
=INFO REPORT==== 6-Jan-2020::13:22:29.108957 ===                   
    alarm_handler: {set,{{disk_almost_full,"/Volumes/InstallESD"},[]}}                                                                 
=INFO REPORT==== 6-Jan-2020::13:22:29.112036 ===                   
    alarm_handler: {set,{system_memory_high_watermark,[]}}                                                                             
[info] 2020-01-06T21:22:29.332120Z nonode@nohost <0.251.0> -------- Preflight check: Asserting Admin Account                           
                                                                   
[info] 2020-01-06T21:22:29.332942Z nonode@nohost <0.251.0> --------                                                                    
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%      
  No Admin Account Found, aborting startup.                        
  Please configure an admin account in your local.ini file.        
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%      
                                                                   
=CRASH REPORT==== 6-Jan-2020::13:22:29.834060 ===                  
  crasher:                                                         
    initial call: application_master:init/4                        

***SNIP***

=INFO REPORT==== 6-Jan-2020::13:22:29.846247 ===                   
    application: crypto                                                                                                                                                                                                                                                       
    exited: stopped                                                
    type: temporary                                                                                                                    
{error,{couch,{admin_account_required,{couch_app,start,                                                                                
                                                 [normal,[]]}}}}                                                                                                                                                                                                              
3>                                                                                                                                                                                                                                                                            

In particular, when starting CouchDB through an Erlang shell or another app when we don't have admins defined, we'll get the clear error: {error,{couch,{admin_account_required,{couch_app,start,[normal,[]]}}}}.

Alternatively, we can take a more aggressive approach to eliminate the kernel failure and crash dump, by changing the throw(admin_account_required) error to be erlang:halt(1) which causes the VM to halt immediately. Here's sample output of what the halt approach looks like:

(! 2553)-> ./rel/couchdb/bin/couchdb 
[info] 2020-01-06T21:32:50.714402Z couchdb@127.0.0.1 <0.259.0> -------- Preflight check: Asserting Admin Account

[info] 2020-01-06T21:32:50.714461Z couchdb@127.0.0.1 <0.259.0> -------- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  No Admin Account Found, aborting startup.                  
  Please configure an admin account in your local.ini file.  
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

and the Erlang shell based approach (note that this kills the shell):

(! 2554)-> erl -pa rel/couchdb/lib/*/ebin
Erlang/OTP 22 [erts-10.5.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Eshell V10.5.1  (abort with ^G)
1> application:ensure_all_started(couch).
=INFO REPORT==== 6-Jan-2020::13:39:48.845608 ===
    alarm_handler: {set,{{disk_almost_full,"/Volumes/InstallESD"},[]}}
=INFO REPORT==== 6-Jan-2020::13:39:48.849207 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}
[info] 2020-01-06T21:39:49.061816Z nonode@nohost <0.250.0> -------- Preflight check: Asserting Admin Account

[info] 2020-01-06T21:39:49.062572Z nonode@nohost <0.250.0> -------- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  No Admin Account Found, aborting startup.                  
  Please configure an admin account in your local.ini file.  
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

IMO throwing a proper error tuple is the "correct" Erlang approach in terms of properly utilizing the API and allowing for indirect use of CouchDB as an Erlang library. The halt based approach hides some of the details from users at the expense of behaving badly in certain situations. Pros and cons.

[1] http://erlang.org/doc/apps/kernel/application.html#start-1
[2]

couchdb/src/couch/src/couch_sup.erl

Line 102 in 0cdd5f0

throw(admin_account_required);

[janl]

IMO throwing a proper error tuple is the "correct" Erlang approach in terms of properly utilizing the API and allowing for indirect use of CouchDB as an Erlang library. The halt based approach hides some of the details from users at the expense of behaving badly in certain situations. Pros and cons.

How about setting another env var in bin/couchdb[.cmd] COUCHDB_NOT_EMBEDDED=1 or some such, and do the halt solution, which looks most user friendly only if that env var is set, which should not be the case if somebody embeds CouchDB?

[kocolosk]

+1 for halting the VM by default.

Embedded mode is a fun topic and I'm not opposed to a flag that changes the error behavior (would suggest COUCHDB_EMBEDDED_MODE=0 to avoid the double-negative) but I also think there's a longer list of work items for us to really tackle that goal.

[wohali]

My $0.02: COUCHDB_EMBEDDED=1

PR is going up with the halt solution, and am also +1 on a solution to work around this for embedding if anyone wants, it's low hanging fruit; I'll open a new ticket on that.