SaaS Connector Templates

[galvana]

Is your feature request related to a specific problem?

The way we've been releasing SaaS connectors up until now is to include a sample SaaS config and dataset in the Fidesops repo. We've thought about these artifacts as examples which the user can use to create their own SaaS connections. The way this happens now is not very elegant, requiring the user to take these examples and make the necessary API calls to create a connection, register the SaaS config, and register the dataset. Ideally, the SaaS connector artifacts should be "pre-registered" and the user should be able to just create a new instance of a SaaS connector.

Describe the solution you'd like

There are a few things needed for this to happen but it begins with introducing the idea of a "SaaS connector template" which is made up of the following:

• SaaS config
• Dataset
• Custom Python overrides for endpoints Python overrides for SaaS config endpoints #815
• Icons (or other assets)

Fidesops should:

• Maintain a registry of existing connector templates
  • Associate the SaaS config, dataset, and other artifacts as belonging to a unique SaaS connector type
• Update any existing SaaS configs and SaaS datasets in the database on system startup, if the version in the registry is newer than the SaaS config version in the connection config table
  • Need to standardize the version number that already exists in the SaaS configs, they are all currently set to version 0.0.1

A user should be able to:

• Create a new ConnectionConfig and associated entities from a SaaS connector template

Additional dev notes:

The example SaaS configs and datasets currently follow the naming convention of salesforce_connector_example for the fides_key. We should change the value to <instance_fides_key> wherever this occurs and replace it with the user-provided fides_key for the connection config before persisting the SaaS config and dataset to the database.

We also want a central location to track the available SaaS connectors and where their assets are stored, for example:

saas_connector_registry.toml

[mailchimp]
config = /data/saas/mailchimp_config.yaml
custom_functions = /data/saas/mailchimp_functions.py
dataset = /data/saas/mailchimp_dataset.yaml
icon = /data/saas/mailchimp_icon.png

This allows us to add or remove available connectors via configuration or to point to different locations to source the files for a connector template.

Additional Information

• limiting the scope of this ticket for just Ethyca-built connectors, not ones that someone else builds
• no changes needed to the way icons currently work, out of scope for this iteration

How templates relate to instantiation of connection configs

The current manual workflow is as follows

1. Create a connection config

PATCH {{host}}/connection/

[{"name": "Stripe Connection",
    "key": "{{connection_id}}",
    "connection_type": "saas",
    "access": "read"
}]

2. Add a SaaS config to the connection config

PATCH {{host}}/connection/{{connection_id}}/saas_config

{
  "fides_key": "{{connection_id}}",
  "name": "Mailchimp SaaS Config",
  "type": "mailchimp",
  "description": "A sample schema representing the Mailchimp connector for Fidesops",
  "version": "0.0.1",
  ...

3. Add secrets to the connection config

PUT {{host}}/connection/{{connection_id}}/secret

{
    "domain": "{{mailchimp_domain}}",
    "username": "{{mailchimp_username}}",
    "api_key": "{{mailchimp_api_key}}"
}

4. Associate a dataset with the SaaS/connection config

PATCH {{host}}/connection/{{saas_key}}/dataset

[
   {
      "fides_key":"mailchimp_connector_example",
      "name":"Mailchimp Dataset",
      "description":"A sample dataset representing the Mailchimp connector for Fidesops",
      "collections":[
         {
...

The new workflow would be:

1. Create a connection config using data the user input in the UI
2. Lookup the SaaS config in the template registry (by connector type) and replace all occurrences of <instance_fides_key> in the SaaS config with the user provided connection_id/fides_key before storing it in the database (plus the new name and description)
3. Store the secrets (like before)
4. Lookup the Dataset in the template registry (by connector type) and replace all occurrences of <instance_fides_key> in the dataset with the user provided connection_id/fides_key before storing it in the database

We can reuse the existing assign_placeholders util to replace the <instance_fides_key> with the user-provided value.

[eastandwestwind]

This looks good @galvana ! I appreciate the fact that you've detailed the current workflow, too, in contrast with the new one.

My main question is: for the new workflow, will we simply use the existing PATCH {{host}}/connection/, and the backend will handle the saas config / dataset / secret associations based on if we detect the connection type is a pre-registered Saas connector? Or will we use a new endpoint?

If we used the existing endpoint, we'd need to alter it to accept a fides_key for datasets/ saas configs, right?

[pattisdr]

Thanks for the details @galvana. I am still unclear on what the template is? Is it a combination of your saas_connector_registry.toml to define the necessary files, and then the files themselves?

[pattisdr]

Adrian said the template was the collection of relevant files, and then they're defined in the registry -