DBFS (Databricks) Support

[Guichaguri]

Databricks has it's own file system format.

I've read a bit of the DVC code and I think I can work on a PR adding support for it by using its REST API. Is there any reason not to work on it?

[efiop]

Hi @Guichaguri !

I'm not familiar with DBFS, but I would double check if maybe it supports one of the already supported APIs (e.g. we have s3 support, ssh, hdfs etc). From quickly looking into the docs, seems like it is based on s3 somewhere internally, but I can't see any common APIs that we could use from the list that we already support. So looks like a new remote type would indeed be necessary.

Do you use databricks yourself, or do you just want to contribute the support for it?

[majidaldo]

Well they have their dbutils.fs in Python. I think that would be a bit more 'native'. I'm using databricks.

[omrihar]

Hello,
is there any update regarding this issue? I'm also using databricks and would like to try using dvc together with it.

Thanks,
Omri

[skshetry]

@omrihar, unfortunately, no one has worked on it yet. We are more than happy to accept the contribution for the feature.

[majidaldo]

Besides the mapping, I don't know how this would work when using databricks notebooks; there is no git integration.

[omrihar]

Unfortunately I'm not sure I am able to add the databricks support - I'm not well versed enough in databricks and dvc for that.

@majidaldo I was thinking of using databricks DBFS for data storage (as S3 is used). You are correct that the current git integration is very lacking... However it still may be possible to use the dvc API to get the files from a databricks notebook with data lying somewhere in dbfs, maybe?

I'm quite new to databricks and am not yet 100% sure what's the best strategy to deal with the various aspects of solid, full-blown ML workflow on databricks.

[majidaldo]

Unfortunately I'm not sure I am able to add the databricks support - I'm not well versed enough in databricks and dvc for that.

@majidaldo I was thinking of using databricks DBFS for data storage (as S3 is used). You are correct that the current git integration is very lacking... However it still may be possible to use the dvc API to get the files from a databricks notebook with data lying somewhere in dbfs, maybe?

I'm quite new to databricks and am not yet 100% sure what's the best strategy to deal with the various aspects of solid, full-blown ML workflow on databricks.

If you're just getting and uploading files from dbfs, dvc isn't adding value here. The dbfs api and dbfs mount is pretty straightforward to use.
There is the ugly hack of copying your git repo to somewhere that databricks can see.

Databricks wants you to use MLFlow. After using Databricks for a while, I almost despise any platform that throws me into notebook-land.

[isidentical]

We are rapidly moving towards fsspec, and it comes bundled with a databricks implementation. So if there is nothing on the dbfs that would contradict with the DVC's assumption regarding filesystem-related stuff, DVC might start supporting it after the full-adoption (1-2 months) https://github.com/intake/filesystem_spec/blob/master/fsspec/implementations/dbfs.py

[smartkiwi]

I'd like to ressurect this issue.
Can we clarify what kind of DBFS support for DVC we are taking about - start with requirements and use cases.

I can think about several

• DBFS as a blob storage backend for DVC (accessible from withon Datsbricks cluster, where DBFS is accessible through local FS interface, also from outside Databricks env via REST API).

• • DVC use DBFS as a target FS - i.e. datasets files tracked by DVC are stored on DBFS - exposed to various tools that Databricks provides - I.e. parquet file can be viewed in SQL mode. And there is some mechanism for switching between versions of dataset

I'm considering the way to start using Databricks for ML project workflows. So the second requirement is way important to me.
And I cared way less where DVC stores blobs, as long it is accessible from both within Databricks env and outside, like developers laptop or in container ran by CI pipeline. Such storage can be s3.

For now I'm thinking about using EC2 instance with s3 bucket DBFS mounted and checking out multiple versions of code and datasets into separate folders. So for Databricks tools information abut version will be lost - it would just see multiple folders with files.

I'm also going to try using Databricks Repo feature with git repo that had DVC set up in it - and run DVC commands after git checkout and before git commit.

[edmondop]

Was this resurrected?