Cluster Setup doesn't set consistent admin user

[wohali]

Similar to #1752, if you use the cluster setup wizard to create admin users on new nodes in the cluster, the created admin user will not use an identical salt on every node, meaning Fauxton fails to obtain a valid _session unless you manually distribute the ini file [admins] user yourself using a config management tool.

Expected Behavior

Setup wizard creates a real server admin user that is usable from Fauxton immediately.

Current Behavior

Setup wizard creates an admin user that cannot be load balanced because the salt is different.

Possible Solution

Pass the salt along as part of the cluster setup API endpoint?

/cc @janl

[jaydoane]

Not sure if this is useful, but I wrote a function to set admin passwords that could maybe be used and/or repurposed? The idea is that it takes a while for the first, locally set password to get hashed, so it waits until the hashing occurs, and only then sets the same Username, and HashedPassword on the other nodes in the cluster:

set_cluster_admin_password(Username, Password) ->
    Section = "admins",
    ok = config:set(Section, Username, Password),
    Condition = fun() ->
        case config:get(Section, Username, Password) of
            Password ->
                wait;
            Hashed ->
                {ok, Hashed}
        end
    end,
    {ok, HashedPassword} = test_util:wait(Condition, 100, 5),
    {Results, []} = rpc:multicall(config, set,
        [Section, Username, HashedPassword]),
    true = lists:all(fun(ok) -> true end, Results),
    {Passwords, []} = rpc:multicall(config, get, [Section, Username]),
    true = lists:all(fun(P) -> P =:= HashedPassword end, Passwords),
    HashedPassword.

This would probably need some better error handling to be viable, but it seems to work under normal conditions:

(node1@127.0.0.1)42> couch_util:set_cluster_admin_password("foo", "pass").
"-pbkdf2-588c7472a6289cef771e31da99e02ca5c48829e8,122b938d618a8062b6c1aebcb74d6059,10"

[jaydoane]

I'm not familiar with fauxton: does it use the _config endpoint to configure admin users? If so, it would seem we'd need to expose a new endpoint for set_cluster_admin_password functionality. Is that correct, or is there an existing endpoint that might be adaptable?

[wohali]

It uses the cluster setup endpoints, which do correctly set admins remotely, not via the config endpoint. See https://github.com/apache/couchdb/tree/master/src/setup

They just don't sync salts.

[jaydoane]

To sync salts, we could modify this code something like this:

diff --git a/src/couch/src/couch_passwords.erl b/src/couch/src/couch_passwords.erl
index baf78f5d5..403b057ba 100644
--- a/src/couch/src/couch_passwords.erl
+++ b/src/couch/src/couch_passwords.erl
@@ -41,18 +41,24 @@ hash_admin_password(ClearPassword) when is_binary(ClearPassword) ->
     hash_admin_password(Scheme, ClearPassword).

 hash_admin_password("simple", ClearPassword) -> % deprecated
-    Salt = couch_uuids:random(),
+    Salt = salt(),
     Hash = crypto:hash(sha, <<ClearPassword/binary, Salt/binary>>),
     ?l2b("-hashed-" ++ couch_util:to_hex(Hash) ++ "," ++ ?b2l(Salt));
 hash_admin_password("pbkdf2", ClearPassword) ->
     Iterations = config:get("couch_httpd_auth", "iterations", "10000"),
-    Salt = couch_uuids:random(),
+    Salt = salt(),
     DerivedKey = couch_passwords:pbkdf2(couch_util:to_binary(ClearPassword),
                                         Salt ,list_to_integer(Iterations)),
     ?l2b("-pbkdf2-" ++ ?b2l(DerivedKey) ++ ","
         ++ ?b2l(Salt) ++ ","
         ++ Iterations).

+salt() ->
+    case config:get("couch_httpd_auth", "salt") of
+        undefined -> couch_uuids:random();
+        Salt -> list_to_binary(Salt)
+    end.
+
 -spec get_unhashed_admins() -> list().
 get_unhashed_admins() ->
     lists:filter(

and then fauton could be modified to obtain a uuid from a node

$ curl -u adm:pass http://127.0.0.1:15984/_uuids
{"uuids":["32528dddde8aac395b7dc6848b000a78"]}

and then set the config option on each node separately via

$ curl -u adm:pass http://127.0.0.1:15984/_node/node1@127.0.0.1/_config/couch_httpd_auth/salt -X PUT -d '"32528dddde8aac395b7dc6848b000a78"'

after which the way it's currently setting the admin passwords should start yielding the same hashes. I might be worried that having the salt in the config file is a security concern, but then I notice that we already store the salt with each admin user already

(node1@127.0.0.1)17> config:get("admins", "admin", "pass").
"-pbkdf2-54d94fc58cabe34b3cf6391c71290dd0198e2996,32528dddde8aac395b7dc6848b000a78,10"

[wohali]

@jaydoane But that breaks the entire cluster setup wizard approach, doesn't it?

[jaydoane]

@wohali Hmm, I thought it kept the current approach mostly the same, and just added a couple additional requests beforehand to sync the salt on all nodes. In what way do you think it breaks the setup wizard approach? I suppose salt could be added to the setup request, but then it would mean we'd need to be careful to apply that before setting any admin users get set. And fauton would still need to either request or generate its own salt so it's the same on all nodes.

[wohali]

@jaydoane setup wizard can be fully run from the CLI. does your workflow work WITHOUT Fauxton and WITHOUT additional API calls? that's the point of the setup wizard.

See http://docs.couchdb.org/en/latest/setup/cluster.html#the-cluster-setup-api

I know for a fact people use this in Chef recipes et. al. and expect it to do all the setup for them.

This ticket was specifically to fix that API endpoint so it did all the things...

[jaydoane]

@wohali ok, got it -- thanks for patiently explaining the requirements. It seems like your initial suggestion of passing the salt in with the enable_cluster action, in combination with making the salt configurable in couch_passwords could work. Would it be an error if a user passes in both password_hash (a current option) and password_salt, or do we just give one precedence over the other? Would we require a minimum length or randomness for the salt or just use whatever gets passed in? Can you think of other subtleties?

[wohali]

Ideally I'd like it to set the admin password on the "coordinator" node, then re-use that salt on subsequent nodes added to the cluster. This way you'd never have to pass in the salt.

[jaydoane]

It seems that we can sync the salt from the coordinator node at the end of setup, because it's only then, after the other nodes have been added, that all nodes will become connected. Similarly, we can sync the admin users from the coordinating node at the same time. It turns out these operations are pretty much independent, and syncing the salt is only really necessary for subsequently added admin users to have the same salt and hashed values (which still seems valuable).

In any case, I have a working proof of concept which still needs some error handling before it's worthy of review.

[janl]

I had this fixed a while ago in this commit: apache/couchdb-setup@54623ce

The scenario is first-time setup: the admin us created on the coordinating note, after that, only the password hash which includes the salt, is sent to the other nodes.

I’m not sure how this plays with adding new nodes to existing clusters, but that’s so far been out of scope for the setup wizard.