Multi Organizations/Users setup

[ceckoslab]

We would like to experiment and find out how Basic RUM can work with more than one Organizations. Today we assume that Basic RUM is used by exactly one organization and this will be happening for the majority of the cases what if more than one organizations would like to use the same Basic RUM backoffice?

I can't be sure what is the right way for implementing this but we need to start from somewhere, so below is the proposal:

Proposal:

We should be able to define a list of users/organizations that are using a given instance of Basic RUM.

For this let's use a configuration file that will be placed in the same folder from where we run the binary of front_basicrum_go. Let's call this file users-map.yaml.

An example of the file will look like:

--- # Multi-user configuration for Basic RUM
- user:
    prefix: fashion_bg
    brum_trace: BY83kR5MS5PM
    expire: 2023-12-25

- user:
    prefix: sofia_news
    brum_trace: ENPu8ctP6zda
    expire: 2023-11-22 

We will have file watcher that reads this file every minute. This will allow us to add new users without restart the Front Basic RUM component.

Database:

Front Basic RUM must create a new table for each new user added in the file. This will be happening on runtime so no restart of Front Basic RUM is required.

We also should account for schema migrations. If we add new columns Basic RUM we should be automatically make migrations for all user tables.

What each configuration filed means:

prefix

Prefix will be used as prefix for the ClickHouse table that the user will be assigned to. Also this prefix will be used as the folder name where we will be doing backups on the file system.

trace

Each user will be given an unique identifier. This identifier will be present on the beacons that will be send by Boomerang JS. The beacon parameter from where we will be reading the trace will be brum_trace. After we read the beacon brum_trace we will be comparing it with the brum_trace we configured in users-map.yaml. If there is a match we will be persisting the data for the respective user to the CluckHouse table and backup folder.

If there is no match we will be just doing a backup of the data to a folder called unknown. We don't need to store this data in ClickHouse.

expire

We will be checking the configured expire date:

1. If the check passes then we will be persisting the data for the respective user to the CluckHouse table and backup folder.
2. If the check does not pass we will be just doing a backup of the data to a folder called expired. We don't need to store this data in ClickHouse.

[ivan-stankov-golang]

I think having table per user is not manageable in the future.

What do you think about table hostname_by_user:

• hostname
• user_id

edit variable hostname query to:

select hostname from hostname_by_user where user_id = ${__user.id}

https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#__user

Think about it and let's discuss it.

[ceckoslab]

@ivan-stankov-golang great proposal and thanks for suggesting this. I wasn't thinking about this possibility while writing the ticket.

I think that we need to validate the proposal in order to be sure if it will work in our case:

If we go with this idea we will need to extend the ClickHouse partition key for our tables. More about the partition key here: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/custom-partitioning-key

Also do you think that we could use Org ${__org} instead of user? I think that in the future it will make sens the data to be partitioned and filtered by organization instead of user.

More about the Org variable here: https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#__org

[ivan-stankov-golang]

If you use organization you need to import the same dashboards and re-create the datasources again:
https://grafana.com/docs/grafana/latest/administration/organization-management/#about-organizations

[ivan-stankov-golang]

use one table for store events.
store backups for all host in the save files - not one per user. TODO
self register hostname administrator -> multiple host
1 host -> 1 API KEY
hostname administrator can register new users
hostname administrator can give access to host to own users

Users are filtered to see only their hosts.

Need to show hostname filter based on logged in grafana user.

Use case

1. User register on grafana website (hostname administrator)
2. Create a new hostname - we generate API KEY, expiration date time
3. User adds the API KEY in Wordpress/Magento/custom plugin
4. Client send beacons with API KEY in new parameter. TODO: pick parameter name.
5. front_basicrum_go receives the beacon.
6. Validate the API KEY. If not valid and archive enable then save as unknown
7. If API KEY is valid, but expired and archive is enabled then save as expired
8. if archive is disabled - ignore beacon. TODO: log ?

Use case: add second hostname

• in grafana create a new hostname - we generate API KEY, expiration date time

Use case: add second user

• in grafana - Create new user

Use case: Add hostname to user

• you must be hostname administrator (custom role)
• in grafana - pick a user from your list
• add a host from your list

TODO check:

• grafana self register
• grafana develop custom plugin
• grafana oauth provider support
• if grafana UI integration is not possible - then lookup custom UI
• check grafana hostname filter security - Is it possible to edit client side e.g. javascript/html
• or we need to add the hostmaping table in dashboard queries

[ivan-stankov-golang]

https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/github/

[ceckoslab]

@ivan-stankov-golang in order to identify from which user/team we get beacon data, I suggest we to identify this by reading a beacon parameter that identifies the user/team.

Let's call for now this parameter subscription_id. We can have a discussion about different naming conventions as well.

[ivan-stankov-golang]

https://volkovlabs.io/blog/how-to-connect-the-data-manipulation-plugin-for-grafana-to-api-server-1abe5f60c904/

[ivan-stankov-golang]

oauth providers (TODO: @ceckoslab)

• google
• github
• facebook ?

or manually/automatically created username/password

• hide login/password from login page
• create alternative login page with only username/password