Support server <> server replication

[tahpot]

See draft spec: https://github.com/verida/VIPs/blob/develop/VIPs/vip-6.md#research-next-steps

• Research next steps
• Finalise spec
• Implement in storage node
• Implement in SDK See [client-ts] Support VIP6 (server side replication) verida-js#272
• Write passing unit tests

[tahpot]

Research next steps

Confirm the role based data access assumptions are correct

Open Fauxton on database A (dev1):

1. Create a user with the role test-replicater (password123)
2. Create a test database test-replication-database
3. Create security entry in the test database granting test-replicater role access as a member
4. Create a validate_doc_update() entry that allows test-replicater role to write

{
  "_id": "_design/only_permit_write_users",
  "validate_doc_update": "\n    function(newDoc, oldDoc, userCtx, secObj) {\n        if ([\"v6c116e6212a63f80eedc231d6b9e2473f7e07e15ee2959b0ce6f22ee76742c41\"].indexOf(userCtx.name) == -1 && userCtx.roles.indexOf('test-replicater') == -1) throw({ unauthorized: 'User is not permitted to write to database' });\n}"
}

Note: The addition to the normal code is && userCtx.roles.indexOf('test-replicater') == -1

5. Login to Fauxton with user test-replicater and confirm it can read and write the test-replication-database.
6. Create a JWT auth token for user test-replicater and confirm it can read and write the test-replication-database.

Confirm push replication works as expected

Open Fauxton on database B (dev2):

1. Create a user with the role test-replicater
2. Create a JWT auth token for the user that never expires

Open Fauxton on database A (dev1):

1. Create a replication entry in database A that replicates to database B and configure it to authenticate with the JWT auth token

{
    "_id": "serverA-replication-",
    "source": "http://myserver.com/foo",
    "target": {
        "url": "http://localhost:5984/bar",
        "auth": {
            "basic": {
                "username": "user",
                "password": "pass"
            }
        }
    },
    "create_target":  true,
    "continuous": true
}

2. Create a new record on database A

Open Fauxton on database B (dev2):

1. Verify the new record was created
2. Verify the validate_doc_update() entry has been replicated
3. Verify the _security entry has been replicated and includes the role test-replicater as a member

Test pad

curl -H "Host: acacia-dev1.tn.verida.tech:5984"
-H "accept: application/json, text/plain, /"
-H "authorization: Bearer "
"https://acacia-dev2.tn.verida.tech/_session"

curl -H "Host: acacia-dev1.tn.verida.tech:5984"
-H "accept: application/json, text/plain, /"
-H "authorization: Bearer "
"https://acacia-dev1.tn.verida.tech/test-replication-database/_all_docs?include_docs=true"

[tahpot]

Testing has proven that Auth JWT's can not be used for authentication between storage nodes. The spec has been updated to reflect this (use username/password instead).

[tahpot]

Implementation tasks:

The following high level changes need to be made to the Storage Node and Client SDK.

Storage node

Endpoints:

1. /auth/replicationCreds - Generate credentials for replicating storage nodes
2. /user/checkReplication - Confirm replication is configured correctly for; all databases OR a single database

Other changes:

1. components/dbManager/configurePermissions write and delete functions need to check userCtx.roles includes the role ${didContextHash}-replicator
2. components/dbManager/configurePermissions security doc needs to add the role ${didContextHash}-replicator in the members list
3. Refactor the storage of databases in a context to have their own database so they can be syncronized via CouchDB replication

Client SDK

1. Call checkReplication() when opening a context to ensure the replication is working as expected.
2. Don't start using a new node until it is fully syncronized? (need to check /status)
3. Open a connection to a random (active) endpoint, rather than all the endpoints