{CI} Fix Content-Length header's value in recordings

[jiasli]

Description

An alternative solution for #20533 to pass the check introduced by Azure/azure-sdk-for-python#20888. This time fix Content-Length header's value in recordings instead.

Failed tests

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1216795&view=logs&j=771f6cdf-8d98-5d9b-2fb9-d3a6e9554788&t=0a182f1d-9946-580d-467e-4d4ebd6af32d&l=15587

2021-11-29T04:22:07.3247612Z =========================== short test summary info ============================
2021-11-29T04:22:07.3249776Z FAILED src/azure-cli/azure/cli/command_modules/monitor/tests/latest/test_monitor_metric_alert_scenarios.py::MonitorTests::test_metric_alert_v2_scenario
2021-11-29T04:22:07.3251399Z FAILED src/azure-cli/azure/cli/command_modules/network/tests/latest/test_private_endpoint_commands.py::NetworkPrivateLinkWebappScenarioTest::test_private_link_resource_webapp
2021-11-29T04:22:07.3253024Z FAILED src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_decorator.py::AKSUpdateDecoratorTestCase::test_update_default_mc_profile
2021-11-29T04:22:07.3254400Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_functionapp_commands.py::FunctionAppWithConsumptionPlanE2ETest::test_functionapp_consumption_e2e
2021-11-29T04:22:07.3255835Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_functionapp_commands.py::FunctionAppWithConsumptionPlanE2ETest::test_functionapp_consumption_ragrs_storage_e2e
2021-11-29T04:22:07.3257109Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_functionapp_commands.py::FunctionappLocalContextScenarioTest::test_functionapp_local_context
2021-11-29T04:22:07.3258333Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_functionapp_commands.py::FunctionappNetworkConnectionTests::test_functionapp_vnetE2E
2021-11-29T04:22:07.3259643Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_logicapp_commands.py::LogicappBasicE2ETest::test_logicapp_e2e
2021-11-29T04:22:07.3261243Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_add
2021-11-29T04:22:07.3262595Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_add_http_header
2021-11-29T04:22:07.3264086Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_add_ip_address_validation
2021-11-29T04:22:07.3265567Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_add_scm
2021-11-29T04:22:07.3266888Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_add_service_endpoint
2021-11-29T04:22:07.3268434Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_add_service_tag_validation
2021-11-29T04:22:07.3269793Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_mixed_remove
2021-11-29T04:22:07.3271272Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_remove
2021-11-29T04:22:07.3272785Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_remove_scm
2021-11-29T04:22:07.3274291Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_set_complex
2021-11-29T04:22:07.3275940Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_set_simple
2021-11-29T04:22:07.3277220Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_show
2021-11-29T04:22:07.3278653Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_access_restriction_commands.py::WebAppAccessRestrictionScenarioTest::test_webapp_access_restriction_slot
2021-11-29T04:22:07.3279745Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappBasicE2ETest::test_webapp_e2e
2021-11-29T04:22:07.3280802Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappQuickCreateTest::test_create_in_different_group
2021-11-29T04:22:07.3281889Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappQuickCreateTest::test_linux_webapp_quick_create_cd
2021-11-29T04:22:07.3282966Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappQuickCreateTest::test_win_webapp_quick_create
2021-11-29T04:22:07.3284039Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappQuickCreateTest::test_win_webapp_quick_create_cd
2021-11-29T04:22:07.3285304Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappQuickCreateTest::test_win_webapp_quick_create_runtime
2021-11-29T04:22:07.3286434Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::BackupWithName::test_backup_with_name
2021-11-29T04:22:07.3287644Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::AppServiceLogTest::test_download_win_web_log
2021-11-29T04:22:07.3288718Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::AppServicePlanScenarioTest::test_auto_delete_plan
2021-11-29T04:22:07.3289806Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::AppServicePlanScenarioTest::test_retain_plan
2021-11-29T04:22:07.3290886Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappConfigureTest::test_update_webapp_settings_thru_json
2021-11-29T04:22:07.3292100Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappConfigureTest::test_webapp_config
2021-11-29T04:22:07.3293345Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappConfigureTest::test_webapp_config_appsettings
2021-11-29T04:22:07.3294493Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappConfigureTest::test_webapp_update_site_configs_persists_ip_restrictions
2021-11-29T04:22:07.3295616Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::LinuxWebappScenarioTest::test_linux_webapp
2021-11-29T04:22:07.3296735Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::LinuxWebappRemoteSSHScenarioTest::test_linux_webapp_remote_ssh
2021-11-29T04:22:07.3298051Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappACRScenarioTest::test_acr_integration
2021-11-29T04:22:07.3299155Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappGitScenarioTest::test_webapp_git
2021-11-29T04:22:07.3300463Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappSlotScenarioTest::test_webapp_slot
2021-11-29T04:22:07.3301973Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappSlotTrafficRouting::test_traffic_routing
2021-11-29T04:22:07.3303071Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::AppServiceCors::test_webapp_cors
2021-11-29T04:22:07.3304213Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappSlotSwapScenarioTest::test_webapp_slot_swap
2021-11-29T04:22:07.3305345Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappSSLCertTest::test_webapp_ssl
2021-11-29T04:22:07.3308535Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappSSLImportCertTest::test_webapp_ssl_import
2021-11-29T04:22:07.3310146Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappSSLImportCertTest::test_webapp_ssl_import_crossrg
2021-11-29T04:22:07.3311321Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappUndeleteTest::test_webapp_deleted_list
2021-11-29T04:22:07.3312432Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappAuthenticationTest::test_webapp_authentication
2021-11-29T04:22:07.3313488Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappUpdateTest::test_webapp_update
2021-11-29T04:22:07.3314665Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappZipDeployScenarioTest::test_deploy_zip
2021-11-29T04:22:07.3315799Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappImplictIdentityTest::test_webapp_assign_system_identity
2021-11-29T04:22:07.3317115Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::ContainerWebappE2ETest::test_container_webapp_docker_image_name
2021-11-29T04:22:07.3318455Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::ContainerWebappE2ETest::test_container_webapp_long_server_url
2021-11-29T04:22:07.3319570Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_create_vnetE2E
2021-11-29T04:22:07.3320681Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_create_with_vnet_by_subnet_rid
2021-11-29T04:22:07.3322019Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_create_with_vnet_by_vnet_rid
2021-11-29T04:22:07.3323150Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_hybridconnectionE2E
2021-11-29T04:22:07.3324262Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_vnetDelegation
2021-11-29T04:22:07.3325728Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_vnetE2E
2021-11-29T04:22:07.3326865Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_vnetRouteAll
2021-11-29T04:22:07.3327950Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_vnetSameName
2021-11-29T04:22:07.3329076Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappNetworkConnectionTests::test_webapp_vnetSubnetId
2021-11-29T04:22:07.3330180Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappDeploymentLogsScenarioTest::test_webapp_list_deployment_logs
2021-11-29T04:22:07.3331328Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappDeploymentLogsScenarioTest::test_webapp_show_deployment_logs
2021-11-29T04:22:07.3332437Z FAILED src/azure-cli/azure/cli/command_modules/appservice/tests/latest/test_webapp_commands.py::WebappLocalContextScenarioTest::test_webapp_local_context

[jiasli]

The script to fix Content-Length header:

import yaml
import glob
import os

affected_modules = ['appservice']

command_modules_path = r'D:\cli\azure-cli\src\azure-cli\azure\cli\command_modules'

yaml_files = []

for module in affected_modules:
    module_path = os.path.join(command_modules_path, module)
    glob_result = glob.glob(os.path.join(command_modules_path, module, "**/*.yaml"), recursive=True)
    yaml_files.extend(glob_result)


yaml_files += [
    r'D:\cli\azure-cli\src\azure-cli\azure\cli\command_modules\monitor\tests\latest\recordings\test_metric_alert_v2_scenario.yaml',
    r'D:\cli\azure-cli\src\azure-cli\azure\cli\command_modules\network\tests\latest\recordings\test_private_link_resource_webapp.yaml'
]

print("YAML files to patch:")
print('\n'.join(yaml_files))

# exit(0)

for yaml_file in yaml_files:
    print("Fixing file:", yaml_file)

    # Read from file
    with open(yaml_file) as f:
        content = yaml.safe_load(f)

    for transaction in content['interactions']:
        print('  Fixing request:', transaction['request']['uri'])
        if 'content-length' not in transaction['response']['headers']:
            print('No Content-Length, skipping')
            continue

        literal_len = transaction['response']['headers']['content-length'][0]
        actual_len = len(transaction['response']['body']['string'])

        print(f'    Literal Content-Length: {literal_len}; Actual Content-Length: {actual_len}')
        transaction['response']['headers']['content-length'][0] = str(actual_len)

    # Save to file
    with open(yaml_file, 'w') as f:
        yaml.dump(content, f, Dumper=yaml.CDumper)

Note: CDumper must be used instead of Dumper, otherwise different result is generated compared to vcrpy (yaml/pyyaml#590).

[StrawnSC]

@jiasli does this script have to be run on the recording files every time tests are re-recorded, or will the Content-Length be correct for future recordings?

[jiasli]

Every time you re-record tests, the Content-Length header will be fixed automatically. See #20668.