json path query results shouldn't have trailing \r

[colemickens]

imgid="$(az image show -n "${d}" -g "${rg}" --q "[id]" -o tsv)"
az vm create ... "${imgid}"

results in:

az vm create ... $'/subs.../snip.vhd\r'

Hard to imagine how this is helpful for anyone. It'd be nice if I didn't have to run these types of commands through tr just to use them in scripts.

[tjprescott]

What OS/shell are you running on?

[colemickens]

The driving shell is Bash 4.4.23 under Linux. The actual az commands are being run from the official docker container, however.

[colemickens]

(alternatively, maybe there's a better way for me to get at these sort of values? I have sort of copy/pasted this -o tsv -q ... pattern for a long time, maybe there's a better option.)

[tjprescott]

That's pretty much the pattern. -o tsv --query id. I don't see that behavior when I query a resource ID, but you mention it is specific to a path. @adewaleo since this involves the az image command, can you take a look?

[adewaleo]

@colemickens, @tjprescott

I tried this out and the command succeeded at the CLI level, (failed to deploy , likely because the test image I used wasn't generalized).

$ ID="$(az image show -n "test-image" -g ova-test --query "[id]" -o tsv)"
$ echo $ID
/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/ova-test/providers/Microsoft.Compute/images/test-image
$az vm create -n test-vm-from-image --image $ID -g ova-test
Deployment failed. Correlation ID: 0000000000000000000000. {
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "OSProvisioningTimedOut",
        "message": "OS Provisioning for VM 'test-vm-from-image' did not finish in the allotted time. However, the VM guest agent was detected running. This suggests the guest OS has not been properly prepared to be used as a VM image (with CreateOption=FromImage). To resolve this issue, either use the VHD as is with CreateOption=Attach or prepare it properly for use as an image:\r\n * Instructions for Windows: https://azure.microsoft.com/documentation/articles/virtual-machines-windows-upload-image/ \r\n * Instructions for Linux: https://azure.microsoft.com/documentation/articles/virtual-machines-linux-capture-image/ "
      }
    ]
  }
}

What error does the CLI raise, if any? I imagine it shouldn't matter ebcause the shell removes whitespace from the command before passing it along to the CLI.

P.S.

I can verify that -o tsv does add a newline in its output (which makes sense as with multiple returned values - i.e. list output - there is a new line separator between each tab-separated row). But as explained above, this shouldn't be an issue.

$ az image show -n "test-image" -g ova-test --query "[id]" -o tsv | xxd
. . .
00000060: 436f 6d70 7574 652f 696d 6167 6573 2f74  Compute/images/t
00000070: 6573 742d 696d 6167 650a                 est-image.

[colemickens]

I made a minimal repro: https://github.com/colemickens/azure-cli-8348-repro

There are two commits, the first commit has log.txt showing the \r attached to the image name, and something throwing back a 400 (I'm not sure what the CLI does, if it resolves the image reference locally and constructs that URL improperly... or if it's maybe CRP erroring on an invalid image ID (due to the trailing \r)?

The second commit adds | tr -d '\r' which resolves the issue and makes the pair of scripts work as expected.

[adewaleo]

Hello @colemickens, I can look into your repro examples. Thanks for putting the effort to do so. I just have some questions

What is your CLI version (guessing less than 2.0.52) and your OS (guessing windows)? I ask because a similar bug was fixed in version 2.0.52. See this PR and this closed issue.

az --version shows your CLI version.

[colemickens]

In that repo, az.sh wraps the Docker container, which should be on latest now, I pull regularly.

Also, I can confirm that the Azure CLI version being used on my dev machine is newer:

$ ./az.sh --version

+ sudo docker run -it -v /nix:/nix -v /tmp/azure-cli:/tmp/azure-cli -e AZURE_CONFIG_DIR=/tmp/azure-cli -e AZURE_STORAGE_CONNECTION_STRING= microsoft/azure-cli:latest az --version
azure-cli (2.0.55)

acr (2.1.12)
acs (2.3.14)
advisor (2.0.0)
ams (0.3.2)
appservice (0.2.11)
backup (1.2.1)
batch (3.4.1)
batchai (0.4.6)
billing (0.2.0)
botservice (0.1.4)
cdn (0.2.0)
cloud (2.1.0)
cognitiveservices (0.2.4)
command-modules-nspkg (2.0.2)
configure (2.0.20)
consumption (0.4.2)
container (0.3.11)
core (2.0.55)
cosmosdb (0.2.7)
dla (0.2.3)
dls (0.1.7)
dms (0.1.1)
eventgrid (0.2.0)
eventhubs (0.3.2)
extension (0.2.3)
feedback (2.1.4)
find (0.2.13)
hdinsight (0.2.0)
interactive (0.4.1)
iot (0.3.4)
iotcentral (0.1.5)
keyvault (2.2.9)
lab (0.1.5)
maps (0.3.3)
monitor (0.2.8)
network (2.3.0)
nspkg (3.0.3)
policyinsights (0.1.0)
profile (2.1.2)
rdbms (0.3.5)
redis (0.3.2)
relay (0.1.2)
reservations (0.4.1)
resource (2.1.8)
role (2.3.0)
search (0.1.1)
security (0.1.0)
servicebus (0.3.2)
servicefabric (0.1.11)
signalr (1.0.0)
sql (2.1.7)
storage (2.3.0)
telemetry (1.0.0)
vm (2.2.12)

Python location '/usr/local/bin/python'
Extensions directory '/tmp/azure-cli/cliextensions'

Python (Linux) 3.6.4 (default, Jan 10 2018, 05:20:21) 
[GCC 6.4.0]

Legal docs and information: aka.ms/AzureCliLegal

I pulled some bits out of the log from the repro repo:

• �[36mCommand arguments: ['vm', 'create', '--debug', '--name', 'nixos-test-26081', '-g', 'nixos-user-westus2', '--image', '/subscriptions/aff271ee-e9be-4441-b9bb-42f5af4cbaeb/resourceGroups/nixos/providers/Microsoft.Compute/images/nixos-image-16.09.1694.019dcc3-x86_64-linux.vhd\r', '--os-disk-size-gb', '50', '--admin-username', 'azureuser', '--ssh-key-value', 'ssh-rsa SNIPPED', '--size', 'Standard_D32s_v3', '--location', 'westus2', '--subscription', 'aff271ee-e9be-4441-b9bb-42f5af4cbaeb']�[0m

---

Actually, here it is. Looks like the CLI is querying for images and includes the '\r' literal.

�[36mmsrest.http_logger : Request URL: 'https://management.azure.com/subscriptions/aff271ee-e9be-4441-b9bb-42f5af4cbaeb/resourceGroups/nixos/providers/Microsoft.Compute/images/nixos-image-16.09.1694.019dcc3-x86_64-linux.vhd%0D?api-version=2018-10-01'�[0m

It would be good to either have another way of isolating single important values from an API response, or for TSV to skip emitting the final trailing '\r'. This case alone proves that it is not prudent to rely on tools to handle extra whitespace chars and other tools would likely have similar problems.

Final note: the repro repo requires nothing to run. If you run ./az.sh login and export AZURE_SUBSCRIPTION_ID=... then they are entirely idempotent and will create all necessary resources to demonstrate the issue occurring.

[colemickens]

Interesting! Maybe this is because of my use of Docker and not related to CLI behavior? moby/moby#8513 But then again, you showed that you could see the \r in your CLI output via hex editor too... Hm. Anyway, thanks for looking into this! (edit: for anyone reading this in the future, I think the az output includes an LF, but docker/moby is translating it to CRLF due to some Linux/tty behaviors).

[colemickens]

Yeah, if I drop -t from the Docker invocation, it looks like it resolves all of this. It looks like it was indeed due to my use of Docker + circumstances described in this issue: moby/moby#8513

Sorry for the distraction @adewaleo and thanks again for looking at this.