Include Signed-urls in the External Tools Framework

[raprasad]

In a March 16th meeting with the Dataverse team, improvements to the external tools framework were discussed. Specifically, the team described an external tools improvement to provide signed urls instead of passing a Dataverse general API token. In transitioning from Dataverse to an external tool, Dataverse would:

• Dynamically generate signed URLs that grant limited-time access to specific API endpoints. The characteristics of these signed URLs will be defined in an updated external tools manifest/specification.
• Pass these signed urls to the external tool via a POST request to:
  • Avoid having any form of authorization token appear in the browser history.
  • Minimize potential exposure and malicious use of the signed urls, in places including server logs. (Server log settings for the DPcreator should not expose POST data.)

These signed urls would have the following characteristics:

(a) Limited in scope and linked to a particular user

• The signed urls would be limited in scope. Examples include:
  • A url to retrieve a Dataverse Datafile would be limited to a particular file. In addition, the user connected to the url will be checked to make sure they/she/he still have permissions for the operation.
  • Similarly, a url to retrieve Schema.org JSON-LD dataset info would be limited to a specific dataset and user permissions would be checked.
  • If it’s important to also support rich clients, the signed URL generated by DataVerse could use a registered mime-type. (@joshua-oss note)

(b) Limited in time

• As specified in a prior manifest or equivalent, each signed url passed to the external tool would have to be used within a certain time window. Examples include:
  • Retrieving a file within 30 minutes or
  • Depositing DP release files within 48 hours.

(c) Encoded with a Dataverse signature

• URL creation note: The URLs are always created in the context of an authenticated user session and the signature links to the users. (@joshua-oss note)
• Each signed url would contain a cryptographic signing token that allows Dataverse to confirm that the URL hasn't been changed. Additional tokens (signed as part of the URL) identify:
  • The time the signing was done and
  • The user it was done for, allowing Dataverse to securely verify that the URL is being used within proper (a) * scope and (b) time. (* that the user connected to the signed url still has permissions to make the request specified by the URL)

(d) Involve Usage tracking

• Each time the signed url is used, the Dataverse would log usage information including:

  • Timestamp. Datetime, timezone.
  • Identifying information. Identifying information related to the request source (such as IP address).
  • Success. Whether the request was successful or not. An example of an invalid request may be one where a retrieval failed due to a timeout or missing/corrupt/deleted data.
  • Validity. Whether the request was valid or invalid. Examples of invalid requests:
    • The signed-url timeframe has expired;
    • The user for whom the signed url was created no longer has permissions for the operation
    • etc.

  This was not specifically addressed in the meeting but in addition to the signed urls, the external tools framework should still include defining and passing along the data currently specified as “queryParameters” and described by the Reserved Words in the external tools documentation. Note: The exception is the “apiGeneralToken”--this would be replaced by the specialized urls.

---

• Reference to orig doc -- limited access

[djbrooke]

• Relevant doc: https://docs.google.com/document/d/1rDhL2QBY2NhVqan3Mwp11BJ0O9naiFEhvnlbKVDQkBc/edit
• The SP team has requested some changes to external tools (re: Globus). Both are part of the framework but this is related to how information is exchanged, and Globus changes are likely related to entry point and tool scope
• In scope - tools will take a POST vs a GET
• Note the logging section - we expect this will be covered by current logging, but we should check
• We should generate a test scenario in case the OpenDP tool is not available. We should discuss with @raprasad to see if there's a way to test that we're missing. We can see the POST in the browser to see if it failed and this may be enough to test.

[qqmyers]

Could potentially be approached as ~3 sub-tasks:

• enable tools to be called via POST
• enable sending signed URLs to tools
• enable authentication/validation of signed API calls

Relevant code snippets: UrlSignerUtil.java has sign/validate methods, #6957 is an old PR where Dataverse calls javascript to do a POST in response to a button push, and changes in #7568 (which add a workflow invocationId as a short-lived, workflow-specific alternative to an APIKey) highlight one option for where you could validate the URLs (i.e. you'd fail to findAuthenticatedUser(), which is used across the API, if the URL was invalid).

[raprasad]

Condensed security notes from @joshua-oss:

Since this is authorizing access to resources, on a time-bound basis, it would be good to examine:

The authorization token generation process

• The description says there is a private key, so that mitigates many potential attacks. The focus then is on the protection of the private key, including the signing process that needs to load the private key into memory. The degree of hardening here will depend heavily on the threat model (e.g. “adversary exploits an XSS attack to trick the signer into signing a fraudulent URI” might be out of scope).

How the system identifies resources

• Based on the description, I’m also assuming that resource identification is done via URIs. It’s always good to make sure that the system is carefully handling name resolution, escaping, internationalization in a way that doesn’t allow minting of URIs that refer to something that’s already been minted.

How the system defines time to ensure that none of these can be spoofed or otherwise controverted

• I’m guessing you already have an expiring token library in mind, so item (b) may be adopting best practices. But it’s always good to think through how the technique handles things like people messing with clocks.

[raprasad]

Desired functionality:

• Will there be a way to test if a signed url is still valid w/o using it?

Use case from the DPcreator perspective:

• User resumes an in-progress workflow.
  • The user originally logged in via Dataverse which generated new signed urls
  • However, this time the user logs in directly to the DPcreator and has access to the original signed urls held in dpCreator
• Can we test that the signed urls are still valid w/o actually using them?
  • e.g. Can we see if the download file signed url is still valid w/o actually downloading the file?
  • Note: This is a proxy for checking whether the user still has permissions for working with the file
• Is there a better way to check if a signed url is valid w/o exercising it or an existing alternative?
  • Some type of expiration date? (Although perms may have changed on the DV side)

---

{
   "signedUrls":[
      {
         "type":"fileDownload",
         "signedUrl":"https://some-signed-url-343234-234324",
         "expirationDate":"2021-05-12T13:40:06Z"
      },
      {
         "type":"metdataRetrievalJSONLD",
         "signedUrl":"https://some-signed-url-343234-234324",
         "expirationDate":"2021-05-12T13:40:06Z"
      }
   ]
}

[qqmyers]

Here's a description of how the signedURLs I proposed work, which I think addresses the questions that have been asked (if I missed some, let me know).

Unless I made a typo, here's an example of a signedURL that should be validate-able against the apikey d4ef4fd5-891e-4ef5-9a1c-e05e3aff1d60 and a never-shared secret key that would be stored within Dataverse (79cc5072-a157-4111-b84e-2fe4a0c93638 in this case) I made this URL manually but followed the logic in the URLSignerUtil class I've mentioned:

https://demo.dataverse.org/api/access/datafile/1827764?format=original&gbrecs=true&until=2021-05-13T17:09:42.411&user=method=GET&token=52e7d0b918ebfd64a12ff03728b9788f664f994d38ca0b6f3b3a14c2f97c2af4db409ee1cc942992019abfae5b6611b28633bd5f72877649e323ea9252032f83
Looking at the params - the format=original and gbrecs=true are part of the base URL I chose to sign (this is an unrestricted file in a public dataset on the demo server so the URL will work, not because demo is validating (it doesn't), just because the extra params for signing are ignored).

• The until= param lets you see the date the URL is valid until
• The method param shows you this is a GET call
• There is an optional user= param (not shown here) that would give you the user id. This would be used in the OpenDP case.

Given that these are all visible, the receiver (OpenDP) can inspect them and, for example, verify that the url is still valid and/or get the user id. (The validation mechanism, as discussed below, doesn't require keeping the date/time or user id secret and having them visible in the URL doesn't help anyone trying to create a malicious URL.)

When the URL is used, the server (Dataverse in the OpenDP case) looks at the user= param, gets the apikey for that user and then creates a sha512 hash of the string
https://demo.dataverse.org/api/access/datafile/1827764?format=original&gbrecs=true&until=2021-05-13T17:09:42.411&user=method=GET&token=79cc5072-a157-4111-b84e-2fe4a0c93638d4ef4fd5-891e-4ef5-9a1c-e05e3aff1d60

where the last param is the concatenation of the never-shared secret Dataverse key and the api key of the listed user (also not shared with OpenDP via the proposed external tool api changes, but potentially available from other sources).

If, and only if, that sha512 hash matches the original token= param sent on the URL (as it should here if I manually created the URL correctly - you can try it at https://emn178.github.io/online-tools/sha512.html for example), the server (Dataverse) can be assured that the URL has not been changed since it was signed. If that's true, Dataverse can then check:

• the date/time (since Dataverse created the date/time initially, and Dataverse is the one checking to see if the date/time means the URL is expired, we know the check is being done by the same (Dataverse machine) clock, so there's no way to cheat if the receiver's clock is deliberately shifted). If the use is after the time listed, it fails.
• the method - if the param doesn't match the HTTP method actually used it fails.

Dataverse can then use the user= param (which has to match with the api key used to originally sign because that user id was also used by Dataverse to choose the right key for validation) to assign the user's privileges to this api call.

Any change to the URL results in the hash failing and Dataverse would then reject the call as not authorized.

If the user's permissions in Dataverse have been revoked since the signedURL was created, the call would still fail. (There isn't a way from the URL to know if that has happened, but one could potentially use a Dataverse admin api (with a credential supplied out-of-band) to query Dataverse about that.)

To create a valid signedURL outside Dataverse would require knowing the appropriate user's APIkey (possible available from, for example, the user's browser unless we change Dataverse to not ever show/distribute API keys once signed URLs are implemented) AND the never-shared secret Dataverse key.

The latter (which doesn't exist yet since URLSigning has so far only been used for Dataverse requesting data from other services) could be stored on disk with access limited to the account running Dataverse, or, if there's concern that someone could still read it, this key could be stored in a service such as Vault (not recommending this - just an example if storing the key with restricted access on the Dataverse server is not secure enough. Given that a user who could get to a key stored that way could probably do other harmful things to Dataverse, just protecting the Dataverse server as well as possible might be the best option.)

To prevent someone from request many URLs to derive the secret Dataverse key, it would also make sense to regenerate that key periodically. That could be manual, or automated.

(One aspect of this model with a time-limited signed URL, that wouldn't be true with for example, a one-time use URL, is that there is no database record in Dataverse related to the signed URL (there could be logging of the creation of the signedURL, but validation of the URL doesn't require looking up any per-signed-URL database entry. This helps with scalability and makes things simpler as Dataverse doesn't need an extra table or to worry about removing entries from that table if the URLs are never used, etc.)

[raprasad]

@qqmyers, thank you for the detailed explanation ^

[djbrooke]

• enable tools to be called via POST could be rolled out separately, mainly for integrators (enable sending signed URLs to tools and enable authentication/validation of signed API calls will likely have to go together)
• we should reach out to integrators and external tool makers about this and encourage them to update, but we should keep the old way for some period of time (we should revisit the autogeneration of tokens for users, but note that the tokens are generated but only a hash is sent)
• to use this new mechanism we'd have to update the previewers (for ex.) to receive POST
• consider the implications for installations that want to serve sensitive data - the old way may not be useful for sensitive data situations

[mreekie]

sprint

• (Len) Waiting on a conversation with Raman and the team
• That conversation may lead to the development of a standalone tool that would mean that we would not have to store L3 or higher data on dataverse
• Leave this in WIP for now.
• Post meeting determine what's next for this.