az cli logs sensitive info in --DEBUG mode

[harneetvirk]

Related command

az cli logs sensitive info in --DEBUG mode

Is your feature request related to a problem? Please describe.

Run "az ml data create --debug --file some_dataset.yaml" with your default workspace and resourcegroup.
Check the debug output. It contains something like this:

azure.core.pipeline.policies._universal: {
"secretsType": "AccountKey",
"key": "2ww2c/dCC42vx********************"
}

Describe the solution you'd like

No secrets are getting exposed from any of the az commands when executing in DEBUG mode.

[yonzhan]

@jiasli for awareness

[jiasli]

I am not aware of az data. Are you developing your own extension?

---

Also, what version of Azure CLI you are using? The HTTP log should now be logged by cli.azure.cli.core.sdk.policies, instead of azure.core.pipeline.policies._universal.

---

As for sensitive information shown in debug log, I think this is a common issue.

az storage commands once face the same issue. #10473 disabled HTTP trace for Track 1 SDK:

azure-cli/src/azure-cli/azure/cli/command_modules/storage/_client_factory.py

Lines 144 to 147 in 93bbc71

	logger.debug('Disable HTTP logging to avoid having storage keys in debug logs')
	client = storage_client_factory(cli_ctx)
	client.config.enable_http_logger = False
	return client.storage_accounts

but client.config.enable_http_logger = False was removed during Track 2 migration (#15845)

azure-cli/src/azure-cli/azure/cli/command_modules/storage/_client_factory.py

Lines 148 to 150 in b40afb9

	logger.debug('Disable HTTP logging to avoid having storage keys in debug logs')
	client = storage_client_factory(cli_ctx)
	return client.storage_accounts

making keys appear again in the debug log:

> az storage account keys list -n myst --debug
...
cli.azure.cli.core.sdk.policies: {"keys":[{"creationTime":"2021-09-24T05:54:33.0930905Z","keyName":"key1","value":"HzdZdX6...","permissions":"FULL"},{"creationTime":"2021-09-24T05:54:33.0930905Z","keyName":"key2","value":"Ht1KOxV...","permissions":"FULL"}]}

Other commands such as az redis list-keys also log keys in the debug log.

[harneetvirk]

Sorry, there was a typo in the description of the issue. This is not a new extension, but we are using ML extension.

az ml data create --name mytestdata01 --file ./src/azure-ai-ml/tests/test_configs/dataset/data_local_path.yaml --version 1 --debug

azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.MachineLearningServices/workspaces/REDACTED/datastores/workspaceblobstore/listSecrets?api-version=2022-05-01'
Request method: 'POST'
Request headers:
'Accept': 'application/json'
'x-ms-client-request-id': 'REDACTED'
'User-Agent': 'azureml-cli-v2/0.0.139 azure-ai-ml/0.0.139 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.7.10 (Windows-10-10.0.22621-SP0)'
'Authorization': 'Bearer REDACTED'
Request body:
This request has no body
urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.MachineLearningServices/workspaces/REDACTED/datastores/workspaceblobstore/listSecrets?api-version=2022-05-01 HTTP/1.1" 200 None
azure.core.pipeline.policies._universal: Response status: '200'
Response headers:
'Cache-Control': 'no-cache'
'Pragma': 'no-cache'
'Transfer-Encoding': 'chunked'
'Content-Type': 'application/json; charset=utf-8'
'Content-Encoding': 'gzip'
'Expires': '-1'
'Vary': 'Accept-Encoding'
'x-ms-ratelimit-remaining-subscription-writes': '1199'
'Request-Context': 'appId=cid-v1:REDACTED'
'x-ms-response-type': 'standard'
'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
'X-Content-Type-Options': 'nosniff'
'x-aml-cluster': 'vienna-eastus-01'
'x-request-time': '0.242'
'Server-Timing': 'traceparent;desc="00-REDACTED"'
'x-ms-request-id': 'c070a7b6-e748-452a-8083-930ce858f8f7'
'x-ms-correlation-request-id': 'REDACTED'
'x-ms-routing-request-id': 'WESTUS:20220926T162712Z:REDACTED'
'Date': 'Mon, 26 Sep 2022 16:27:11 GMT'
Response content:
{
"secretsType": "AccountKey",
"key": "REDACTEDKEY=="
}

[jiasli]

--debug logging sensitive information is by design, as the very purpose of --debug is to print the raw HTTP trace. To prevent unexpected credential leak, please do not use --debug for sensitive commands.

If you can provide strong business justification that we should make --debug mask secrets and introduce additional switch such as --debug-with-secrets for the original behavior, we are very willing to discuss with our PMs about it.

[BalassaMarton]

I'd recommend adding a --debug-no-secrets switch. The business justification is that we will be submitting bug reports for Azure CLI and the issue template requires us to paste full debug output.