Monitor Activity-Log improvements

[JasonRShaver]

This is a WIP task...

Issue 0

How do I get all End Request events for a given resource? It seems I have to make up a --filter 100% from scratch, as it ignores the simple filters.

Issue 1

Issue: az monitor activity-log list needs an example, especially for --filter and --caller.

Issue 2

az monitor activity-log list --end-time 2017-10-27
usage error: [--correlation-id ID | --resource-group NAME | --resource-id ID | --resource-provider PROVIDER]

Issue 3
Change default table output for az monitor activity-log list

Issue 4
An invalid --filter returns a Bad Request, is it possible to identify the filter as the issue?

az monitor activity-log list --resource-group MonitorTest --query '[].{Submission:submissionTimestamp,Caller:caller,Category:category.value,Group:resourceGroupName,Operation:operationName.localizedValue,EventName:eventName.value}' --filter 'eventName==EndRequest'
Operation returned an invalid status code 'Bad Request'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/azure/cli/main.py", line 36, in main
    cmd_result = APPLICATION.execute(args)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/application.py", line 216, in execute
    result = expanded_arg.func(params)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 377, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 630, in _execute_command
    raise client_exception
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 618, in _execute_command
    exception_handler(ex)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/monitor/_exception_handler.py", line 21, in monitor_exception_handler
    reraise(*sys.exc_info())
  File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 602, in _execute_command
    result = op(client, **kwargs) if client else op(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/monitor/custom.py", line 267, in list_activity_log
    return _limit_results(activity_log, max_events)
  File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/monitor/custom.py", line 339, in _limit_results
    for index, item in enumerate(paged):
  File "/usr/local/lib/python3.6/site-packages/msrest/paging.py", line 109, in __next__
    self.advance_page()
  File "/usr/local/lib/python3.6/site-packages/msrest/paging.py", line 95, in advance_page
    self._response = self._get_next(self.next_link)
  File "/usr/local/lib/python3.6/site-packages/azure/monitor/operations/activity_logs_operations.py", line 117, in internal_paging
    raise models.ErrorResponseException(self._deserialize, response)
azure.monitor.models.error_response.ErrorResponseException: Operation returned an invalid status code 'Bad Request'

Issue 5
Can we get this something more clear?
time data '2017-10-27' does not match format '%Y-%m-%dT%H:%M:%SZ'

Issue 6
How do you do an activity-log export with the CLI? This seems like a pretty important command

[meiliumeiliu]

Use command to pull activity log, but ServiceHealth log can't be get.

Command:
az monitor activity-log list --start-time 2018-07-28T00:00:00Z --end-time 2018-07-29T00:00:00Z --resource-group NULL

Expect ServiceHealth log:
{
"channels": "Admin",
"correlationId": "680f7025-f66f-40a4-b2e3-6026ef9c57fc",
"description": "Active: Delays in ARM generated Activity Logs",
"eventDataId": "ec3b9f05-2f30-ce15-438b-472e6eb486c2",
"eventName": {
"value": "",
"localizedValue": ""
},
"category": {
"value": "ServiceHealth",
"localizedValue": "Service Health"
},
"eventTimestamp": "2018-07-28T00:55:09.3204073Z",
"id": "/subscriptions/3ed4ee1c-1a83-4e02-a928-7ff5e0008e8a/events/ec3b9f05-2f30-ce15-438b-472e6eb486c2/ticks/636683361093204073",
"level": "Informational",
"operationId": "",
"operationName": {
"value": "Microsoft.ServiceHealth/incident/action",
"localizedValue": "Microsoft.ServiceHealth/incident/action"
},
"resourceGroupName": "",
"resourceProviderName": {
"value": "",
"localizedValue": ""
},
"resourceType": {
"value": "",
"localizedValue": ""
},
"resourceId": "/subscriptions/3ed4ee1c-1a83-4e02-a928-7ff5e0008e8a",
"status": {
"value": "Active",
"localizedValue": "Active"
},
"subStatus": {
"value": "",
"localizedValue": ""
},
"submissionTimestamp": "2018-07-28T00:55:28.6046196Z",
"subscriptionId": "3ed4ee1c-1a83-4e02-a928-7ff5e0008e8a",
"properties": {
"title": "Delays in ARM generated Activity Logs",
"service": "Activity Logs & Alerts",
"region": "Global",
"communication": "<p style="margin: 0in; margin-bottom: .0001pt">Summary of impact: Between 23:00 UTC on 24 Jul 2018 and 00:30 UTC on 27 Jul 2018, customers may have not received activity logs for ARM resources. All service management operations for ARM resources would have been successful, however the corresponding activity logs would not have been generated. As a result, any dependent actions or workflows that are dependent on ARM generated activity logs would have not been initiated. Engineers have confirmed that customers can receive ARM generated activity logs as of 00:30 on the 27 July 2018. However, at this time customers may not able to access the activity logs that were generated during this impact time frame. 

<p style="margin: 0in; margin-bottom: .0001pt">Preliminary root cause: Engineers determined that a recent Azure Resource Manager deployment task interrupted a data pipeline, preventing proper routing of Activity Log data.  

<p style="margin: 0in; margin-bottom: .0001pt">Mitigation: Engineers deployed a platform hotfix to mitigate the immediate issue. 
Next steps: Engineers are in the process of applying extended mitigation to fully mitigate any additional impact. This incident is going to remain active as engineers access options for the missing logs that would have been generated during the impact window. The next update will be in 72 hours, or as events warrant.

<p style="margin: 0in; margin-bottom: .0001pt">

",
"incidentType": "Incident",
"trackingId": "FLLH-MRG",
"impactStartTime": "2018-07-24T23:00:00.0000000Z",
"impactMitigationTime": "2018-07-27T00:30:00.0000000Z",
"impactedServices": "[{"ImpactedRegions":[{"RegionName":"Global"}],"ServiceName":"Activity Logs & Alerts"}]",
"impactedServicesTableRows": "\r\nActivity Logs & Alerts\r\nGlobal
\r\n\r\n",
"defaultLanguageTitle": "Delays in ARM generated Activity Logs",
"defaultLanguageContent": "<p style="margin: 0in; margin-bottom: .0001pt">Summary of impact: Between 23:00 UTC on 24 Jul 2018 and 00:30 UTC on 27 Jul 2018, customers may have not received activity logs for ARM resources. All service management operations for ARM resources would have been successful, however the corresponding activity logs would not have been generated. As a result, any dependent actions or workflows that are dependent on ARM generated activity logs would have not been initiated. Engineers have confirmed that customers can receive ARM generated activity logs as of 00:30 on the 27 July 2018. However, at this time customers may not able to access the activity logs that were generated during this impact time frame. 

<p style="margin: 0in; margin-bottom: .0001pt">Preliminary root cause: Engineers determined that a recent Azure Resource Manager deployment task interrupted a data pipeline, preventing proper routing of Activity Log data.  

<p style="margin: 0in; margin-bottom: .0001pt">Mitigation: Engineers deployed a platform hotfix to mitigate the immediate issue. 
Next steps: Engineers are in the process of applying extended mitigation to fully mitigate any additional impact. This incident is going to remain active as engineers access options for the missing logs that would have been generated during the impact window. The next update will be in 72 hours, or as events warrant.

<p style="margin: 0in; margin-bottom: .0001pt">

",
"stage": "Active",
"communicationId": "11000005691004",
"version": "0.1.1"
},
"relatedEvents": []
}

[kbroughton]

I'm going to add on to this bug report / improvement as it seems highly related.
I get this result when using --start-time from over 3 months ago presumeably because activity logging retention is set to 3 months I my request was 3 months + delta.
If i change the request to just under 3 months, it works.

$ az monitor activity-log list --resource-group azsanner_vm_rg --start-time 2018-05-20T16:06:27Z
Operation returned an invalid status code 'Bad Request'
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/knack/cli.py", line 194, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 331, in execute
six.reraise(*sys.exc_info())
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 304, in execute
result = cmd(params)
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 168, in call
return super(AzCliCommand, self).call(*args, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/knack/commands.py", line 109, in call
return self.handler(*args, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/azure/cli/core/init.py", line 348, in default_command_handler
result = op(**command_args)
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/azure/cli/command_modules/monitor/operations/activity_log.py", line 47, in list_activity_log
return _limit_results(activity_log, max_events)
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/azure/cli/command_modules/monitor/operations/activity_log.py", line 111, in _limit_results
for index, item in enumerate(paged):
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/msrest/paging.py", line 109, in next
self.advance_page()
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/msrest/paging.py", line 95, in advance_page
self._response = self._get_next(self.next_link)
File "/usr/local/Cellar/azure-cli/2.0.27/libexec/lib/python3.6/site-packages/azure/mgmt/monitor/operations/activity_logs_operations.py", line 118, in internal_paging
raise models.ErrorResponseException(self._deserialize, response)
azure.mgmt.monitor.models.error_response.ErrorResponseException: Operation returned an invalid status code 'Bad Request'

Modified request to < 3 months
$ az monitor activity-log list --resource-group azsanner_vm_rg --start-time 2018-05-21T16:06:27Z

returns data.

Bad Request suggests a syntax error. I think the error should be more accurate or preferrably, the request truncated to < 3 months (default retention) in a try: except:

[tjprescott]

• 0: Yes, you would need to create the filter from scratch. However, it seems like EventName isn't supported by the service. The filter should be something like --filter "eventName/value eq 'EndRequest'" Ultimately, this is a service limitation.
• 1: Add example.
• 2: Fix param set validation.
• 3: Will not fix. There is no way to set per-command default output formats.
• 4: This has been fixed. "BadRequest" now shows the inner error (if available).
• 5: Fix datetime validation.
• 6: Pipe the output to file.