Skip to content

Conversation

@justinmae
Copy link

Note: This is a cherry-pick of #9891 to the v2 branch.

Description

Add support for the --monitoring-configuration parameter to the emr create-cluster command, enabling users to configure CloudWatch logging for EMR clusters through the CLI.

Motivation

This change exposes an existing EMR API feature (MonitoringConfiguration) through the AWS CLI, allowing users to configure cluster-level CloudWatch logging without needing to use the AWS SDK or Console.

Changes

  • Added MONITORING_CONFIGURATION_SCHEMA to argumentschema.py
  • Added MONITORING_CONFIGURATION help text to helptext.py
  • Integrated --monitoring-configuration parameter in createcluster.py
  • Added unit test test_create_cluster_with_monitoring_configuration

Testing

  • Unit tests pass locally
  • Test validates parameter parsing and API call structure
  • No changes to botocore or service models required

Example Usage

aws emr create-cluster \
  --release-label emr-7.11.0 \
  --instance-groups InstanceGroupType=MASTER,InstanceType=m5.xlarge,InstanceCount=1 \
  --monitoring-configuration '{
    "CloudWatchLogConfiguration": {
        "Enabled": true,
        "LogGroupName": "MyLogGroup",
        "LogStreamNamePrefix": "MyPrefix",
        "EncryptionKeyArn": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012",
        "LogTypes": {
            "STEP_LOGS": ["STDOUT", "STDERR"],
            "SPARK_DRIVER": ["STDOUT"],
            "SPARK_EXECUTOR": ["STDERR"]
        }
    }
}'

Add support for the --monitoring-configuration parameter to the
emr create-cluster command. This enables users to configure CloudWatch
logging for EMR clusters through the CLI.

The monitoring configuration allows specification of:
* CloudWatch log group name and log stream prefix
* Encryption key ARN for log encryption
* Log types (STEP_LOGS, SPARK_DRIVER, SPARK_EXECUTOR) mapped to
  output streams (STDOUT, STDERR)

Implementation includes:
* MONITORING_CONFIGURATION_SCHEMA in argumentschema.py
* MONITORING_CONFIGURATION help text in helptext.py
* Parameter integration in createcluster.py
* Unit test test_create_cluster_with_monitoring_configuration

This change adds CLI support for an existing EMR API feature,
requiring no changes to botocore or service models.
@hssyoo hssyoo self-assigned this Dec 3, 2025
Copy link
Contributor

@hssyoo hssyoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🏆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants