diff --git a/content/docs/command-reference/remote/add.md b/content/docs/command-reference/remote/add.md index c0af342bb8..137e9f6132 100644 --- a/content/docs/command-reference/remote/add.md +++ b/content/docs/command-reference/remote/add.md @@ -88,9 +88,9 @@ DVC will determine the [type of remote](#supported-storage-types) based on the The following are the types of remote storage (protocols) supported: -
+ -### Click for Amazon S3 + > 💡 Before adding an S3 remote, be sure to > [Create a Bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html). @@ -109,13 +109,13 @@ Make sure you have the following permissions enabled: `s3:ListBucket`, methods that are performed by DVC (`list_objects_v2` or `list_objects`, `head_object`, `upload_file`, `download_file`, `delete_object`, `copy`). -> See `dvc remote modify` for a full list of S3 parameters. +> See +> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=Amazon-S3#available-parameters-per-storage-type) +> for a full list of S3 parameters. -
+ -
- -### Click for S3-compatible storage + For object storage that supports an S3-compatible API (e.g. [Minio](https://min.io/), @@ -134,16 +134,15 @@ $ dvc remote modify myremote endpointurl \ By default, DVC authenticates using your AWS CLI [configuration](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) (if set). This uses the default AWS credentials file. To use a custom -authentication method, use the parameters described in `dvc remote modify`. +authentication method, use the parameters described in +[`dvc remote modify`](/doc/command-reference/remote/modify?tab=S3-compatible-storage#available-parameters-per-storage-type). Any other S3 parameter can also be set for S3-compatible storage. Whether they're effective depends on each storage platform. -
- -
+ -### Click for Microsoft Azure Blob Storage + ```dvc $ dvc remote add -d myremote azure://mycontainer/path @@ -158,14 +157,13 @@ a Microsoft application. https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential To use a custom authentication method, use the parameters described in -`dvc remote modify`. See some +[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Microsoft-Azure-Blob-Storage#available-parameters-per-storage-type). +See some [examples](/doc/command-reference/remote/modify#example-some-azure-authentication-methods). -
+ -
- -### Click for Google Drive + To start using a GDrive remote, first add it with a [valid URL format](/doc/user-guide/setup-google-drive-remote#url-format). Then @@ -183,7 +181,9 @@ Go to the following link in your browser: Enter verification code: # <- enter resulting code ``` -See `dvc remote modify` for a list of other GDrive parameters, or +See +[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Google-Drive#available-parameters-per-storage-type) +for a list of other GDrive parameters, or [Set up a Google Drive DVC Remote](/doc/user-guide/setup-google-drive-remote) for a full guide on using Google Drive as DVC remote storage. @@ -195,11 +195,9 @@ modified. > Please note our [Privacy Policy (Google APIs)](/doc/user-guide/privacy). -
- -
+ -### Click for Google Cloud Storage + > 💡 Before adding a GC Storage remote, be sure to > [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets). @@ -211,17 +209,16 @@ $ dvc remote add -d myremote gs://mybucket/path By default, DVC expects your GCP CLI is already [configured](https://cloud.google.com/sdk/docs/authorizing). DVC will be using default GCP key file to access Google Cloud Storage. To override some of these -parameters, use the parameters described in `dvc remote modify`. +parameters, use the parameters described in +[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Google-Cloud-Storage#available-parameters-per-storage-type). > Make sure to run `gcloud auth application-default login` unless you use > `GOOGLE_APPLICATION_CREDENTIALS` and/or service account, or other ways to > authenticate. See details [here](https://stackoverflow.com/a/53307505/298182). -
- -
+ -### Click for Aliyun OSS + First you need to set up OSS storage on Aliyun Cloud. Then, use an S3 style URL for OSS storage, and configure the @@ -233,8 +230,8 @@ $ dvc remote modify myremote oss_endpoint endpoint ``` To set key id, key secret and endpoint (or any other OSS parameter), use -`dvc remote modify` as show below. Use the `--local` option to avoid committing -your secrets to Git: +[`dvc remote modify`](/doc/command-reference/remote/modify?tab=Aliyun-OSS#available-parameters-per-storage-type) +as show below. Use the `--local` option to avoid committing your secrets to Git: ```dvc $ dvc remote modify --local myremote oss_key_id 'mykey' @@ -259,17 +256,17 @@ $ export OSS_ACCESS_KEY_SECRET='mysecret' > Uses default key id and key secret when they are not given, which gives read > access to public read bucket and public bucket. -
+ -
- -### Click for SSH + ```dvc $ dvc remote add -d myremote ssh://user@example.com/path ``` -> See `dvc remote modify` for a full list of SSH parameters. +> See +> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=SSH#available-parameters-per-storage-type) +> for a full list of SSH parameters. ⚠️ DVC requires both SSH and SFTP access to work with remote SSH locations. Please check that you are able to connect both ways with tools like `ssh` and @@ -277,11 +274,9 @@ Please check that you are able to connect both ways with tools like `ssh` and > Note that the server's SFTP root might differ from its physical root (`/`). -
- -
+ -### Click for HDFS + ⚠️ Using HDFS with a Hadoop cluster might require additional setup. Our assumption is that the client is set up to use it. Specifically, [`libhdfs`] @@ -297,13 +292,13 @@ API. Read more about it by expanding the WebHDFS section below. $ dvc remote add -d myremote hdfs://user@example.com/path ``` -> See `dvc remote modify` for a full list of HDFS parameters. - -
+> See +> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=HDFS#available-parameters-per-storage-type) +> for a full list of HDFS parameters. -
+ -### Click for WebHDFS + ⚠️ Using WebHDFS requires to enable REST API access in the cluster: set the config property `dfs.webhdfs.enabled` to `true` in `hdfs-site.xml`. @@ -325,25 +320,25 @@ active kerberos session. > `token` contains sensitive user info. Therefore, it's safer to add it with the > `--local` option, so it's written to a Git-ignored config file. -> See `dvc remote modify` for a full list of WebHDFS parameters. +> See +> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=WebHDFS#available-parameters-per-storage-type) +> for a full list of WebHDFS parameters. -
+ -
- -### Click for HTTP + ```dvc $ dvc remote add -d myremote https://example.com/path ``` -> See `dvc remote modify` for a full list of HTTP parameters. - -
+> See +> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=HTTP#available-parameters-per-storage-type) +> for a full list of HTTP parameters. -
+ -### Click for WebDAV + ```dvc $ dvc remote add -d myremote \ @@ -358,13 +353,13 @@ $ dvc remote add -d myremote \ webdavs://example.com/owncloud/remote.php/dav/files/myuser ``` -> See `dvc remote modify` for a full list of WebDAV parameters. - -
+> See +> [`dvc remote modify`](/doc/command-reference/remote/modify?tab=WebDAV#available-parameters-per-storage-type) +> for a full list of WebDAV parameters. -
+ -### Click for local remote + A "local remote" is a directory in the machine's file system. Not to be confused with the `--local` option of `dvc remote` (and other config) commands! @@ -402,7 +397,7 @@ $ cat .dvc/config > Note that `../dvcstore` has been resolved relative to the `.dvc/` dir, > resulting in `../../dvcstore`. -
+ ## Example: Customize an S3 remote diff --git a/content/docs/command-reference/remote/modify.md b/content/docs/command-reference/remote/modify.md index 05e3e093c9..0f707799a5 100644 --- a/content/docs/command-reference/remote/modify.md +++ b/content/docs/command-reference/remote/modify.md @@ -103,9 +103,9 @@ The following config options are available for all remote types: The following are the types of remote storage (protocols) and their config options: -
+ -### Click for Amazon S3 + - `url` - remote location, in the `s3:///` format: @@ -329,11 +329,9 @@ $ dvc push For more on the supported env vars, please see the [boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables) -
+ -
- -### Click for S3-compatible storage + - `endpointurl` - URL to connect to the S3-compatible storage server or service (e.g. [Minio](https://min.io/), @@ -348,11 +346,9 @@ For more on the supported env vars, please see the Any other S3 parameter (see previous section) can also be set for S3-compatible storage. Whether they're effective depends on each storage platform. -
+ -
- -### Click for Microsoft Azure Blob Storage + > If any values given to the parameters below contain sensitive user info, add > them with the `--local` option, so they're written to a Git-ignored config @@ -531,11 +527,9 @@ can propagate from an Azure configuration file (typically managed with `container_name`. The default directory where it will be searched for is `~/.azure` but this can be customized with the `AZURE_CONFIG_DIR` env var. -
+ -
- -### Click for Google Drive + > If any values given to the parameters below contain sensitive user info, add > them with the `--local` option, so they're written to a Git-ignored config @@ -632,11 +626,9 @@ more information. [delegated]: https://developers.google.com/admin-sdk/directory/v1/guides/delegation -
- -
+ -### Click for Google Cloud Storage + > If any values given to the parameters below contain sensitive user info, add > them with the `--local` option, so they're written to a Git-ignored config @@ -679,11 +671,9 @@ set: $ export GOOGLE_APPLICATION_CREDENTIALS='.../project-XXX.json' ``` -
- -
+ -### Click for Aliyun OSS + > If any values given to the parameters below contain sensitive user info, add > them with the `--local` option, so they're written to a Git-ignored config @@ -725,11 +715,9 @@ $ export OSS_ACCESS_KEY_SECRET='mysecret' $ export OSS_ENDPOINT='endpoint' ``` -
+ -
- -### Click for SSH + > If any values given to the parameters below contain sensitive user info, add > them with the `--local` option, so they're written to a Git-ignored config @@ -818,11 +806,9 @@ $ export OSS_ENDPOINT='endpoint' $ dvc remote modify myremote allow_agent false ``` -
- -
+ -### Click for HDFS + 💡 Using a HDFS cluster as remote storage is also supported via the WebHDFS API. Read more about by expanding the WebHDFS section in @@ -852,11 +838,9 @@ Read more about by expanding the WebHDFS section in kerb_ticket /path/to/ticket/cache ``` -
- -
+ -### Click for WebHDFS + 💡 WebHDFS serves as an alternative for using the same remote storage supported by HDFS. Read more about by expanding the WebHDFS section in @@ -941,11 +925,9 @@ by HDFS. Read more about by expanding the WebHDFS section in [webhdfs api]: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Delegation_Token_Operations -
- -
+ -### Click for HTTP + > If any values given to the parameters below contain sensitive user info, add > them with the `--local` option, so they're written to a Git-ignored config @@ -1033,11 +1015,9 @@ by HDFS. Read more about by expanding the WebHDFS section in $ dvc remote modify myremote ssl_verify path/to/ca_bundle.pem ``` -
- -
+ -### Click for WebDAV + > If any values given to the parameters below contain sensitive user info, add > them with the `--local` option, so they're written to a Git-ignored config @@ -1119,7 +1099,9 @@ by HDFS. Read more about by expanding the WebHDFS section in $ dvc remote modify myremote timeout 120 ``` -
+ + + ## Example: Customize an S3 remote