SQL database audit and threat detection commands by jaredmoo · Pull Request #2536 · Azure/azure-cli

jaredmoo · 2017-03-17T00:57:08Z

db group:

Group
    az sql db: Manage databases.

Subgroups:
    audit-policy           : Manage a database's auditing policy.
    replica                : Manage replication between databases.
    threat-policy          : Manage a database's threat detection policy.

Commands:
    copy                   : Creates a copy of an existing database.
    create                 : Creates a database.
    delete                 : Deletes a database or data warehouse.
    list                   : Lists all databases and data warehouses in a server, or all databases
                             in an elastic pool.
    restore                : Creates a new database by restoring from a database backup.
    show                   : Gets a database or data warehouse.
    update                 : Updates a database.

audit-policy subgroup:

(env) D:\git\azure-cli [datasecurity ≡]> az sql db audit-policy -h

Group
    az sql db audit-policy: Manage a database's auditing policy.

Commands:
    show  : Gets a blob auditing policy associated with a database.
    update: Updates a database's auditing policy.

(env) D:\git\azure-cli [datasecurity ≡]> az sql db audit-policy show -h

Command
    az sql db audit-policy show: Gets a blob auditing policy associated with a database.

Arguments
    --name -n           [Required]: Name of the Azure SQL Database.
    --resource-group -g [Required]: Name of resource group. You can configure the default group
                                    using 'az configure --defaults group=<name>'.
    --server -s         [Required]: Name of the Azure SQL server.

Global Arguments
    --debug                       : Increase logging verbosity to show all debug logs.
    --help -h                     : Show this help message and exit.
    --output -o                   : Output format.  Allowed values: json, jsonc, table, tsv.
                                    Default: json.
    --query                       : JMESPath query string. See http://jmespath.org/ for more
                                    information and examples.
    --verbose                     : Increase logging verbosity. Use --debug for full debug logs.
(env) D:\git\azure-cli [datasecurity ≡]> az sql db audit-policy update -h

Command
    az sql db audit-policy update: Updates a database's auditing policy.
        If the policy is being enabled, storage_account or both storage_endpoint and
        storage_account_access_key must be specified.

Arguments
    --name -n           [Required]: Name of the Azure SQL Database.
    --resource-group -g [Required]: Name of resource group. You can configure the default group
                                    using 'az configure --defaults group=<name>'.
    --server -s         [Required]: Name of the Azure SQL server.

Policy Arguments
    --actions                     : List of actions and action groups to audit.
    --retention-days              : The number of days to retain audit logs.
    --state                       : Auditing policy state.

Storage Arguments
    --storage-account             : Name of the storage account.
    --storage-endpoint            : The storage account endpoint.
    --storage-key                 : Access key for the storage account.

Generic Update Arguments
    --add                         : Add an object to a list of objects by specifying a path and key
                                    value pairs.  Example: --add property.listProperty <key=value,
                                    string or JSON string>.
    --remove                      : Remove a property or an element from a list.  Example: --remove
                                    property.list <indexToRemove> OR --remove propertyToRemove.
    --set                         : Update an object by specifying a property path and value to set.
                                    Example: --set property1.property2=<value>.

Global Arguments
    --debug                       : Increase logging verbosity to show all debug logs.
    --help -h                     : Show this help message and exit.
    --output -o                   : Output format.  Allowed values: json, jsonc, table, tsv.
                                    Default: json.
    --query                       : JMESPath query string. See http://jmespath.org/ for more
                                    information and examples.
    --verbose                     : Increase logging verbosity. Use --debug for full debug logs.

Examples
    Enable an auditing policy by specifying storage account name
        az db audit-policy update -g mygroup -s myserver -n mydb --state Enabled --storage-account
        mystorage
    Enable an auditing policy by specifying storage account name and resource group
        az db audit-policy update -g mygroup -s myserver -n mydb --state Enabled --storage-account
        mystorage
    Enable an auditing policy by specifying storage endpoint and key
        az db audit-policy update -g mygroup -s myserver -n mydb --state Enabled --storage-endpoint
        https://mystorage.blob.core.windows.net --storage-key MYKEY==
    Set the list of audit actions
        az db audit-policy update -g mygroup -s myserver -n mydb --actions
        FAILED_DATABASE_AUTHENTICATION_GROUP 'UPDATE on database::mydb by public'
    Add an audit action
        az db audit-policy update -g mygroup -s myserver -n mydb --add auditActionsAndGroups
        FAILED_DATABASE_AUTHENTICATION_GROUP
    Remove an audit action by list index
        az db audit-policy update -g mygroup -s myserver -n mydb --add auditActionsAndGroups 0
    Disable an auditing policy
        az db audit-policy update -g mygroup -s myserver -n mydb --state Disabled

threat-policy subgroup:

(env) D:\git\azure-cli [datasecurity ≡]> az sql db threat-policy  -h

Group
    az sql db threat-policy: Manage a database's threat detection policy.

Commands:
    show  : Gets a Threat Detection policy associated with a database.
    update: Updates a database's threat detection policy.

(env) D:\git\azure-cli [datasecurity ≡]> az sql db threat-policy show -h

Command
    az sql db threat-policy show: Gets a Threat Detection policy associated with a
    database.

Arguments
    --name -n           [Required]: Name of the Azure SQL Database.
    --resource-group -g [Required]: Name of resource group. You can configure the default group
                                    using 'az configure --defaults group=<name>'.
    --server -s         [Required]: Name of the Azure SQL server.

Global Arguments
    --debug                       : Increase logging verbosity to show all debug logs.
    --help -h                     : Show this help message and exit.
    --output -o                   : Output format.  Allowed values: json, jsonc, table, tsv.
                                    Default: json.
    --query                       : JMESPath query string. See http://jmespath.org/ for more
                                    information and examples.
    --verbose                     : Increase logging verbosity. Use --debug for full debug logs.

(env) D:\git\azure-cli [datasecurity ≡]> az sql db threat-policy update -h

Command
    az sql db threat-policy update: Updates a database's threat detection policy.
        If the policy is being enabled, storage_account or both storage_endpoint and
        storage_account_access_key must be specified.

Arguments
    --name -n           [Required]: Name of the Azure SQL Database.
    --resource-group -g [Required]: Name of resource group. You can configure the default group
                                    using 'az configure --defaults group=<name>'.
    --server -s         [Required]: Name of the Azure SQL server.

Notification Arguments
    --email-account-admins        : Whether the alert is sent to the account administrators.
    --email-addresses             : List of email addresses that alerts are sent to.

Policy Arguments
    --disabled-alerts             : List of disabled alerts.
    --retention-days              : The number of days to retain threat detection logs.
    --state                       : Threat detection policy state.

Storage Arguments
    --storage-account             : Name of the storage account.
    --storage-endpoint            : The storage account endpoint.
    --storage-key                 : Access key for the storage account.

Generic Update Arguments
    --add                         : Add an object to a list of objects by specifying a path and key
                                    value pairs.  Example: --add property.listProperty <key=value,
                                    string or JSON string>.
    --remove                      : Remove a property or an element from a list.  Example: --remove
                                    property.list <indexToRemove> OR --remove propertyToRemove.
    --set                         : Update an object by specifying a property path and value to set.
                                    Example: --set property1.property2=<value>.

Global Arguments
    --debug                       : Increase logging verbosity to show all debug logs.
    --help -h                     : Show this help message and exit.
    --output -o                   : Output format.  Allowed values: json, jsonc, table, tsv.
                                    Default: json.
    --query                       : JMESPath query string. See http://jmespath.org/ for more
                                    information and examples.
    --verbose                     : Increase logging verbosity. Use --debug for full debug logs.

Examples
    Enable by specifying storage account name
        az db threat-policy update -g mygroup -s myserver -n mydb --state Enabled
        --storage-account mystorage
    Enable by specifying storage account name and resource group
        az db threat-policy update -g mygroup -s myserver -n mydb --state Enabled
        --storage-account mystorage
    Enable by specifying storage endpoint and key
        az db threat-policy update -g mygroup -s myserver -n mydb --state Enabled
        --storage-endpoint https://mystorage.blob.core.windows.net --storage-key MYKEY==
    Disable a subset of alert types
        az db threat-policy update -g mygroup -s myserver -n mydb --disabled-alerts
        Sql_Injection_Vulnerability Access_Anomaly
    Configure email recipients
        az db threat-policy update -g mygroup -s myserver -n mydb --email-addresses
        me@examlee.com you@example.com --email-account-admins Enabled
    Disable
        az db threat-policy update -g mygroup -s myserver -n mydb --state Disabled

Use ARM lookup instead. This is a lot of custom code, but without it the customer experience is bad.

msftclas · 2017-03-17T00:57:12Z

@jaredmoo,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

codecov-io · 2017-03-17T01:11:56Z

Codecov Report

Merging #2536 into master will increase coverage by 0.19%.
The diff coverage is 87.28%.

@@            Coverage Diff             @@
##           master    #2536      +/-   ##
==========================================
+ Coverage   72.16%   72.35%   +0.19%     
==========================================
  Files         362      363       +1     
  Lines       19798    19942     +144     
  Branches     2920     2943      +23     
==========================================
+ Hits        14288    14430     +142     
- Misses       4593     4596       +3     
+ Partials      917      916       -1

Impacted Files	Coverage Δ
...zure-cli-sql/azure/cli/command_modules/sql/help.py	`100% <100%> (ø)`	⬆️
...ure-cli-sql/azure/cli/command_modules/sql/_util.py	`90.21% <100%> (+0.44%)`	⬆️
...-cli-sql/azure/cli/command_modules/sql/commands.py	`100% <100%> (ø)`	⬆️
...re-cli-sql/azure/cli/command_modules/sql/custom.py	`85.46% <80.28%> (-3.86%)`	⬇️
...re-cli-sql/azure/cli/command_modules/sql/params.py	`96.7% <96.29%> (-0.22%)`	⬇️
src/azure-cli-core/azure/cli/core/_output.py	`61.4% <0%> (-1.17%)`	⬇️
...li-core/azure/cli/core/test_utils/vcr_test_base.py	`73.68% <0%> (-0.62%)`	⬇️
...iner/azure/cli/command_modules/container/custom.py	`23.21% <0%> (-0.43%)`	⬇️
...e-cli-acs/azure/cli/command_modules/acs/_params.py	`69.09% <0%> (ø)`	⬆️
...zure/cli/command_modules/appservice/_validators.py	`100% <0%> (ø)`
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ce8235...055210f. Read the comment docs.

troydai · 2017-03-17T17:15:24Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/_util.py

            factory=self._client_factory,
-            custom_function_op=custom_function_op)
+            custom_function_op=custom_function_op,
+            **kwargs)


Following is my personal preference, so I don't intend to block this PR here. I usually advise avoiding **kwargs if the arguments can be explicit defined. The keyworded arguments are powerful but it also hurts the readability and maintainability. It is not obvious through the glance at the signature to make out what's the additional arguments here. Can we instead of using keyworded arguments define the arguments explicitly?

troydai · 2017-03-17T17:16:08Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/commands.py

+with ServiceGroup(__name__,
+                  get_sql_database_threat_detection_policies_operations,
+                  database_threat_detection_policy_operations) as s:
+    with s.group('sql db threat-detection-policy') as c:


This is a very long command name.

Yes, understood. Does threat-policy make sense?

Sounds good to me.

troydai · 2017-03-17T17:17:12Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/custom.py

+# to determine the account's keys and endpoint. Why isn't this just a command line parameter:
+# because if it was a command line parameter then the customer would need to specify storage
+# resource group just to update some unrelated property, which is annoying and makes no sense to
+# the customer.


troydai

Looks good to me. I left as few comments but none of them are blocking. You can decide whether you want to address them.

troydai · 2017-03-17T17:40:50Z

src/command_modules/azure-cli-sql/tests/test_sql_commands.py

+    def test_sql_db_security_mgmt(self, resource_group, resource_group_2,
+                                  resource_group_location, server,
+                                  storage_account, storage_account_2):
+        database_name = "cliautomationdb01"


Random name?

jaredmoo · 2017-03-17T17:49:43Z

Thanks Troy. I'll take a look. Please don't merge until I get signoff from SQL data security team on the design & implementation.

troydai · 2017-03-17T17:51:43Z

Got it. I'll put a do not merge label. Please take it down when your team OKed this change.

nathannfan · 2017-03-20T23:35:21Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/help.py

+                  storage_account_access_key must be specified.
+    examples:
+        - name: Enable by specifying storage account name
+          text: az db audit-policy update -g mygroup -s myserver -n mydb


Why not az sql db here?

Typo. Well spotted.

yaakoviyun · 2017-03-20T08:42:04Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/commands.py

+    get_sql_database_operations,
+    get_sql_database_blob_auditing_policies_operations,
+    get_sql_database_threat_detection_policies_operations,
+    get_sql_elasticpools_operations,


are those related to the change?

Yes these functions are used below

yaakoviyun · 2017-03-20T08:43:45Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/custom.py

+# resource group just to update some unrelated property, which is annoying and makes no sense to
+# the customer.
+def _find_storage_account(name):
+    resource_type = 'Microsoft.Storage/storageAccounts'


what about classic storage accounts?

Azure CLI doesn't support creating classic storage accounts, so I have no way to create them for automated test. On top of that, there is no python client library, so I would need to handwrite http request - exactly the kind of thing you would want to have automated. The customer can still use classic storage by specifying storage endpoint and key instead of account name, so classic storage isn't totally blocked, just inconvenient. I'm ok with this for the first release. I'm adding explicit checks.

yaakoviyun · 2017-03-20T08:45:39Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/custom.py

+        account_name=storage_account)
+
+    # Get endpoint
+    return account.primary_endpoints.blob  # pylint: disable=no-member


there are storage accounts without blob parameter.
we need to check that and return a meaningful error asking the user to choose a different storage.

Ok. I'm not sure how to create such a storage account, so I'm taking your word for it ;) I tested by replacing primary_endpoints.blob with primary_endpoints.potato and then handling that exception.

yaakoviyun · 2017-03-20T08:47:21Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/custom.py

+    if storage_endpoint is not None:
+        instance.storage_endpoint = storage_endpoint
+    if storage_account is not None:
+        storage_resource_group = _find_storage_account(storage_account)


"_find_storage_account_resource_group"?

yaakoviyun · 2017-03-20T08:49:45Z

src/command_modules/azure-cli-sql/azure/cli/command_modules/sql/custom.py

+        retention_days=None):
+
+    # Apply state
+    if state is not None:


do we have parameters value validation? (e.g. in the "state" case the valid set is "Enabled" and "Disabled")

Yes, it will be validated due to enum_choice_list in params.py

(env) D:\git\azure-cli [datasecurity ≡]> az sql db audit-policy update --state adf
az sql db audit-policy update: error: argument --state: invalid choice: 'adf' (choose from 'Enabled', 'Disabled')

jaredmoo · 2017-03-21T16:52:58Z

@troydai I have now addressed @yaakoviyun 's feedback so you can now merge. (I don't have perms to remove the label).

@yaakoviyun & @nathannfan if you have any more feedback I can do follow-up PR.

Thanks all

Triggered by Azure CLI Extensions Release Pipeline - ADO_BUILD_ID=577770 Last commit: Azure/azure-cli-extensions@b3ca154

jaredmoo added 12 commits March 14, 2017 13:56

Update python sdk dependency

36b820f

Added db audit & td policy show & update

20adafd

Handle newly visible database properties

5a474f5

Implement security test

b389db1

Removed debug print statements

4fccee1

storage-account-access-key -> storage-key

8e48a89

Added custom funcs and params

2078105

FIxed style errors

a8518a7

Trying to use enum_choice_list

04748e2

Added storage helper parameters

369b5c2

flake8 warning

335234c

Removed --storage-resource-group param.

0b05fad

Use ARM lookup instead. This is a lot of custom code, but without it the customer experience is bad.

msftclas added the cla-not-required label Mar 17, 2017

Fixed examples

a244010

jaredmoo mentioned this pull request Mar 17, 2017

[Generic Update] Not able to include equal sign (=) in the value field #2527

Closed

troydai reviewed Mar 17, 2017

View reviewed changes

troydai approved these changes Mar 17, 2017

View reviewed changes

troydai added the do-not-merge label Mar 17, 2017

jaredmoo added 2 commits March 17, 2017 10:59

Made generic_update_command args explicit

411f26b

threat-detection-policy -> threat-policy

0fb8ef2

jaredmoo mentioned this pull request Mar 17, 2017

Sql Import/Export CLI commands and test #2538

Merged

jaredmoo added 2 commits March 20, 2017 09:20

Merge remote-tracking branch 'azure/master' into datasecurity

47a0463

Commented on import name parameter

85cdc83

nathannfan reviewed Mar 20, 2017

View reviewed changes

Corrected examples

1633bbe

yaakoviyun reviewed Mar 21, 2017

View reviewed changes

Error messages for classic storage and blobless storage

5cf8fa3

Fixed example

055210f

troydai removed the do-not-merge label Mar 22, 2017

troydai merged commit a7c9c68 into Azure:master Mar 22, 2017

jaredmoo deleted the datasecurity branch March 22, 2017 03:39

haroldrandom added the cla-not-required label Oct 25, 2019

00Kai0 pushed a commit to 00Kai0/azure-cli that referenced this pull request Apr 7, 2021

[Release] Update index.json for extension [ connectedk8s ] (Azure#2536)

0e8bb5e

Triggered by Azure CLI Extensions Release Pipeline - ADO_BUILD_ID=577770 Last commit: Azure/azure-cli-extensions@b3ca154

Conversation

jaredmoo commented Mar 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msftclas commented Mar 17, 2017

Uh oh!

codecov-io commented Mar 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

troydai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaredmoo commented Mar 17, 2017

Uh oh!

troydai commented Mar 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaredmoo commented Mar 21, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jaredmoo commented Mar 17, 2017 •

edited

Loading

codecov-io commented Mar 17, 2017 •

edited

Loading

troydai commented Mar 17, 2017 •

edited

Loading