Add Dataset APIs to the webserver

[bobbai00]

This PR introduces the APIs to the webserver. The frontend can utilize these APIs to manage and view dataset, the version of dataset, and file under different versions. This PR depends on the ddl and version control file service, see #2369 .

API overviews

The added APIs include:

1. Dataset and Version related APIs:

# create the dataset
POST    /api/dataset/create

# delete datasets given a list of did
POST    /api/dataset/delete

# update dataset's description
POST    /api/dataset/update/description

# update dataset's name
POST    /api/dataset/update/name

# get the dataset info by did
GET     /api/dataset/{did}

# create a new version of a dataset
POST    /api/dataset/{did}/version/create

# retrieve the latest version of a dataset
GET     /api/dataset/{did}/version/latest

# list all version of a dataset 
GET     /api/dataset/{did}/version/list 

# get the file content of a file of certain version of a dataset
GET     /api/dataset/{did}/version/{dvid}/file

# get the root file nodes of a certaion version of a dataset
GET     /api/dataset/{did}/version/{dvid}/rootFileNodes 

2. Dataset Access Control

PUT     /api/access/dataset/grant/{did}/{email}/{privilege} 
GET     /api/access/dataset/list/{did}
GET     /api/access/dataset/owner/{did}
DELETE  /api/access/dataset/revoke/{did}/{email}

Dataset+Version Filesystem Path Design

Each dataset will be stored in directory user-resources/datasets/{did}. Each directory is managed by GitVersionControlLocalFileStorage. Each version is one commit of this repo.

DB Schema change

A minor update to the Dataset table should be made, by:

USE `texera_db`;

ALTER TABLE dataset
MODIFY COLUMN storage_path VARCHAR(512) NOT NULL DEFAULT '';

So that the storage_path of the dataset will be '' if not set during insert. This default value exists because: the storage path is identified by the did, but did is valid only when the record is inserted into Dataset table.

Some Details

1. Concurrency Control of Dataset Modification & Access

A static lock store is initialized in object DatasetResource, it maps from did to ReentrantLock

  val datasetLocks: scala.collection.concurrent.Map[UInteger, ReentrantLock] =
    new scala.collection.concurrent.TrieMap[UInteger, ReentrantLock]()

When createNewDatasetVersion is called, before performing any operation, it first acquire or create the lock for given did:

  private def createNewDatasetVersion(
      ctx: DSLContext,
      did: UInteger,
      uid: UInteger,
      versionName: String,
      multiPart: FormDataMultiPart
  ): Option[DashboardDatasetVersion] = {

    // Acquire or Create the lock for dataset of {did}
    val lock = DatasetResource.datasetLocks.getOrElseUpdate(did, new ReentrantLock())

    if (lock.isLocked) {
      return None
    }
    lock.lock()

And in finally block, the lock is released

finally {
      // Release the lock
      lock.unlock()
    }

Based on the above mechanism, the below questions can be answered:

How to handle Alice create v2 and Bob is reading v1 file/file tree of dataset d1 simultaneously ?

Both requests will succeed. The read will NOT acquire any locks.

How to handle Alice create v2 and Bob create v2 for dataset d1 simultaneously?

One of them will fail. Because either Alice or Bob can get the Lock, then the other one will fail.

How to handle Alice create v2 for dataset d1, and Bob create v2 for dataset d2?

Both requests will succeed. The lock is at dataset level, meaning that one lock per did.

2. Representation of the File tree using JSON

To represent the root file nodes in JSON in order to make frontend able to parse and display the file tree, a FileNodeSerializer is added and plugged into the bootstrap of dropwizard.

public class FileNodeSerializer extends StdSerializer<FileNode> {

  public FileNodeSerializer() {
    this(null);
  }

  public FileNodeSerializer(Class<FileNode> t) {
    super(t);
  }

  @Override
  public void serialize(FileNode value, JsonGenerator gen, SerializerProvider provider) throws IOException {
    gen.writeStartObject();
    gen.writeStringField("path", value.getRelativePath().toString());
    gen.writeBooleanField("isFile", value.isFile());
    if (value.isDirectory()) {
      gen.writeFieldName("children");
      gen.writeStartArray();
      for (FileNode child : value.getChildren()) {
        serialize(child, gen, provider); // Recursively serialize children
      }
      gen.writeEndArray();
    }
    gen.writeEndObject();
  }
}

    // register a new custom module and add the custom serializer into it
    val customSerializerModule = new SimpleModule("CustomSerializers")
    customSerializerModule.addSerializer(classOf[FileNode], new FileNodeSerializer())
    bootstrap.getObjectMapper.registerModule(customSerializerModule)

To give a example, the JSON format of a file tree:

a.csv
1.txt
dir
- dir/1.pdf

will be:

{
    "fileNodes": [
        {
            "path": "a.csv",
            "isFile": true
        },
        {
            "path": "1.txt",
            "isFile": true
        },
        {
            "path": "dir",
            "isFile": false,
            "children": [
                {
                    "path": "dir/1.pdf",
                    "isFile": true
                }
            ]
        }
    ]
}

[aglinxinyuan]

LGTM! Please remove your config file and fix backend test cases.