Add FileService as a standalone microservice, LakeFS+S3 as dataset storage

[bobbai00]

This PR introduces the FileService as another microservice parallel to WorkflowCompilingService, ComputingUnitMaster/Worker, and TexeraWebApplication.

Purpose of the FileService

• We want to improve the performance of our current Git-based Dataset implementation;
• We decide to go with LakeFS + S3, LakeFS for the version control metadata and S3 for data transfer; But LakeFS doesn't have access control layer
• Therefore, we build the FileService, providing
  • all the APIs related to versioned files in datasets
  • access control

Architecture before and after adding FileService

Before:

After

Key Changes

• A new service, FileService is introduced. All the dataset-related endpoints are hosted on FileService
• Several configuration items related to LakeFS and S3 are introduced in the storage-config.yaml
• Frontend UI updates to incorporate with new changes
• For ComputingUnitMaster and ComputingUnitWorker, they will call FileService to read files, during which their access will be verified. So in the dynamic computing architecture (which will be introduced in Add computing unit manager service #3298), they will send requests along with current user's token. In single-machine architecture, they are bypassing the network requests by doing direct local function calls.
• Python UDF can now directly read dataset's file by the following example code:

file = DatasetFileDocument("The URL of the file")
bytes = file.read_file() # return an io.Bytes object

You may refer to core/amber/src/main/python/pytexera/storage/dataset_file_document.py for implementation details. This feature is only available in the dynamic computing architecture.

How to migrate the previous datasets to the new datasets managed by the LakeFS

As we did quite some refactoring, two dataset implementations are NOT compatible with each others. To migrate the previous datasets to the latest implementation, you will need to re-upload the data via the new UI.

How to deploy new architecture

Step1. Deploy LakeFS & Minio

Use Docker (Highly recommended for local development)

• Install Docker Desktop which contains both docker engine and docker compose. Do NOT solely use brew install docker because it will only install docker engine.
• Go to directory: core/file-service/src/main/resources
• Execute docker compose up -d at its directory

Use Binary (Recommended for production deployment)

Refer to https://docs.lakefs.io/howto/deploy/

Step2. Configure the storage-config.yaml

Use Docker

If you deployed lakeFS using docker compose in step 1, you don't need to change this file and can directly proceed with the given default values.

Use Binary

If you deployed lakeFS binary, you need to configure the below section in the storage-config.yaml:

  lakefs:
    endpoint: ""
    auth:
      api-secret: ""
      username: ""
      password: ""
    block-storage:
      type: ""
      bucket-name: ""

  s3:
    endpoint: ""
    auth:
      username: ""
      password: ""

Step3. Launch services

Launch FileService , in addition to TexeraWebApplication, WorkflowCompilingService and ComputingUnitMaster.

Future PRs after this one

• Remove the Dataset related endpoints completely from amber package.
• Incorporate the deployment of LakeFS+S3 in the Helm chart of K8s-based deployment.
• Some optimizations:
  • for small files, directly upload it instead of using multipart upload
  • when doing result exports, use multipart upload as the result size can be quite big.
  • support re-transmit for partially-uploaded files.

[aglinxinyuan]

Since installing LakeFS and Minio can be complex, could we add a frontend flag that allows developers to enable the user system without requiring LakeFS and Minio? This would let developers read files directly from their local file system when the user system is enabled.

[bobbai00]

Since installing LakeFS and Minio can be complex, could we add a frontend flag that allows developers to enable the user system without requiring LakeFS and Minio? This would let developers read files directly from their local file system when the user system is enabled.

OK. I have added a flag in environment.default.ts

[aglinxinyuan]

LGTM!
Tested on both Windows and Mac. The setup is very smooth.
Please add more details on step3. For example, how can developers migrate to this from current master.