Azure Data Factory Extension - Linked Service Creation - not reading JSON file

[codepossible]

Functionality Feedback

When I am trying to run a command to create a linked service where the contents of the properties are stored in JSON file, it is not reading them. It is sending a request to the API with a request body which looks like this -

{"properties": {"type": "LinkedService"}}

Here is the debug log for the request:

azure.core.pipeline.policies._universal : Request URL: 'https://management.azure.com/subscriptions/(redacted-subscription)/resourceGroups/(redacted-rg)/providers/Microsoft.DataFactory/factories/(readacted-adf-name)/linkedservices/HyasDataPrep2?api-version=2018-06-01'
azure.core.pipeline.policies._universal : Request method: 'PUT'
azure.core.pipeline.policies._universal : Request headers:
azure.core.pipeline.policies._universal :     'Content-Type': 'application/json'
azure.core.pipeline.policies._universal :     'Accept': 'application/json'
azure.core.pipeline.policies._universal :     'Content-Length': '41'
azure.core.pipeline.policies._universal :     'x-ms-client-request-id': '28f918d4-5135-11eb-b3a4-0242ac110002'
azure.core.pipeline.policies._universal :     'CommandName': 'datafactory linked-service create'
azure.core.pipeline.policies._universal :     'ParameterSetName': '--factory-name --name --properties --resource-group --debug'
azure.core.pipeline.policies._universal :     'User-Agent': 'AZURECLI/2.14.0 (DEB) azsdk-python-datafactorymanagementclient/unknown Python/3.6.10 (Linux-4.19.128-microsoft-standard-x86_64-with-debian-10.6)'
azure.core.pipeline.policies._universal :     'Authorization': '*****'
azure.core.pipeline.policies._universal : Request body:
azure.core.pipeline.policies._universal : {"properties": {"type": "LinkedService"}}

The az-cli command is -

az datafactory linked-service create --factory-name "redacted-factory-name" --name "linked-service" --properties @./"linked-services/somelinkedservice.json" --resource-group "redacted-rg" --debug

Design Feedback

The JSON payload for the properties is very different what is expressed in ADF Authoring tool, when you look at the JSON declaration on the service.

For example:

• JSON Payload for a Storage account in Portal/Azure Data Factory Authoring tool

{
    "name": "AzureDataLakeStorage1",
    "type": "Microsoft.DataFactory/factories/linkedservices",
    "properties": {
        "annotations": [],
        "type": "AzureBlobFS",
        "typeProperties": {
            "url": "https://astoreageaccountincloud.dfs.core.windows.net",
            "encryptedCredential": "superdupersecretkey"
        }
    }
}

• JSON Payload required by az datafactory linked-service create command

{
 "type": "AzureStorage",
  "typeProperties": {
        "connectionString" :{ 
             "type": "SecureString",
             "value":"DefaultEndpointsProtocol=https;AccountName=examplestorageaccount;AccountKey=<storage key>"
          }
     }
}

Unlike the az datafactory pipeline create command, which seems to accept the same JSON as it is in the pipeline code in the Authoring tool.

I think, for ease of developer experience and consistency, the JSON payload should be same as the one in the portal/authoring tool. . The one in az datafactory pipeline create is the better model. I am hoping the API is consistent as well.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 13386c48-1ac8-9452-fcb3-263829e9982c
• Version Independent ID: 1060fc99-cdb7-9c06-2374-f3127fa5e3f5
• Content: az datafactory linked-service
• Content Source: latest/docs-ref-autogen/ext/datafactory/datafactory/linked-service.yml
• GitHub Login: @rloutlaw
• Microsoft Alias: routlaw

[yonzhan]

datafactory

[yungezz]

thanks for feedback. @fengzhou-msft could you pls have a look? thanks

[fengzhou-msft]

@codepossible for the functionality issue, which platform do you use and how do you install azure-cli? I cannot reproduce it on windows and WSL.

For the design feedback, I think you are comapring 2 differnet types: AzureBlobFS and AzureStorage.

[fengzhou-msft]

For the functionality issue, I can reproduce it with a type that is not supported by the extension yet, for instance: AzureDatabricksDeltaLake and there should be a warning message with --debug:

msrest.serialization: Subtype value AzureDatabricksDeltaLake has no mapping, use base class LinkedService.

Then the request body becomes:
{"properties": {"type": "LinkedService"}}

We will work on regenerating the extension to support new types.