Config Rule

Important note: a new version of the AWS Config console is available! While you can chose to use the new console experience by clicking the link in this image, be aware that the screenshots and guidance in this page reflect the current console. You may need to adjust your steps slightly to accommodate the new user interface.

New Config Console

Create the Lab Environment

Before proceeding, you must create a CloudFormation Stack that includes the resources required for this lab. Click here to deploy this Stack into your account:

Create Stack

Note: This link will take you to the us-east-1 AWS region. If you wish to use another region, you will need to adjust the region in the top right-hand corner of the console.

Note: Make sure to specify the same S3 bucket name in the Bucket parameter as is assigned to the CloudTrail Trail from the previous step.

Creating an AWS Config Rule to Alert on SSM Agent Non-Compliance

In this step we will create an AWS Config rule that will evaluate if EC2 instances have a working AWS Systems Manager Agent.

  1. Go to the AWS Config console, and then click on Rules on the left side of the console.
  2. Click on Add Rule
  3. In the Add Rule screen in the Filter section type ec2-instance-managed-by-systems-manager, click on the ec2-instance-managed-by-systems-manager rule.
  4. Under the Trigger Section take notice of the trigger type, and ensure Configuration Change is selected. Leave the remaining settings as-is.
  5. Click Save

You can create config Rules to monitor a number of items within your infrastructure. Beside utilizing AWS managed Config rules you can also create custom rules using Lambda Functions. Located here in Github are same sample config rules you can create and implement in AWS Lambda.

Deploy an EC2 Instance

Next, let’s deploy and EC2 instance to test our AWS Config rule.

You Can Deploy via Web Console, or you can run the following Command from the AWS CLI. If launching via the Web Console, create a t2.micro Amazon Linux 2 instance. We will not need to access the server itself, so the keypair used is not important

Note that we are not assigning an IAM role to the instance - that comes later!

aws ec2 run-instances --image-id $(aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 --query 'Parameters[0].[Value]' --output text) --count 1 --instance-type t3.large --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=configruletest}]' 

The instance should be up and running in approximately one minute.

Now return to the AWS Config rule you created, click into the rule, and click Re-evaluate after the instance is up and running. You will have wait a minute or two for the result, and then refresh the web page. After a few moments the instance we deployed should be flagged as non-compliant.

Next let’s fix this non-compliant resource by adding a remediation action to the AWS Config rule.

  1. Go back to AWS Config, and edit the ec2-instance-managed-by-systems-manager rule. We will set a Remediation Action to attach a required IAM Role. Under Choose remediation action do the following:

    • Remediation action: AWS-AttachIAMToInstance
    • Resource ID parameter: InstanceId
      • This passes the non-compliant instance ID to the remediation action
    • Get the IAM Role name from the output of the CloudFormation Stack. The parameter is named EC2RoleName. Enter this into the RoleName field.
  2. Click Save

  1. Go back into the AWS Config rule and look at non-compliant resources. Select the instance we deployed and then click on Remediate.

Remediate Button

  1. Visit the AWS Systems Manager console, and then click on Automation on the left side. You should see an Automation Task begin, and this will attach the IAM Role to the instance.

Systems Manager Automation

  1. Once completed, reboot the instance to quicken the process.
  2. Return to the AWS Systems Manager console, and then check under Managed instances. When the instance shows up as a managed instance, re-evaluate the rule AWS Config Rule once more. You will see that the instance is now compliant.
What did we learn?
  • How to create an AWS Config Rule to evaluate if instances are managed by SSM
  • How to use AWS Systems Manager Automation Documents to remediate non-compliant instances

Create an AWS Config Rule to Ensure AWS CloudTrail is Enabled

In this step we will create an AWS Config rule using an AWS managed rule that will evaluate whether AWS CloudTrail is enabled within your AWS account.

  1. Let’s go to the AWS Config console, once there click on Rules in the left side of the console.
  2. Click on Add Rule
  3. In the Add Rule screen, in the Filter section, type cloudtrail-enabled, and then click on the cloudTrail-enabled rule.
  4. Under the Trigger section, notice the trigger type is Periodic.
    • Change the Frequency to 1 hour
  5. We will need to lookup the Amazon Resource Name (ARN) for the CloudWatch logs group that our CloudTrail trail is delivering to. In the CloudWatch console, click on Log groups, and then click on the management-tools-week log group. The ARN will be on the top-left side of the console. Copy this, and paste it into the cloudWatchLogsLogGroupArn field.

CloudWatch Logs Group ARN

  1. Click Save.

When this rule evaluates it will mark your AWS account as compliant, so long as CloudTrail was set up in the previous step properly.

Set Triggers for Lambda Functions

Now we will create the trigger for the Lambda function deployed by our CloudFormation template. This function will be executed as soon as we make our CloudTrail configuration non-compliant!

This is to demonstrate that while we certainly can use AWS Config to perform remediatations, you can choose to use other tools to do so as well. In this case, a CloudWatch event will capture the change notification from AWS Config and trigger a Lambda execution to perform the remediation for us.

Note: This step needs to be done correctly for the Lambda to trigger.

  1. Go to Amazon CloudWatch console, and under Events on the left side click on Rules

    • Click Create rule
    • Under Event Source
      • Select the radio button next to Event Pattern
      • Service Name: Config
      • Event Type: Config Rules Compliance Change
      • Select the radio button next to Specific message type
      • From the Drop Down Select ComplianceChangeNotification
      • Select radio button next to Specific rule name
      • Type cloudtrail-enabled
    • Click Add target
    • Select the ConfigSSMLab-EnforceCloudTrailFunction Lambda function, which is the function deployed by CloudFormation. Feel Free to take a look at the function code in Lambda.

  1. Click Configure details

  2. Enter CloudTrailChange as the rule name, leave the state enabled and then click Create rule

Testing the Enforce CloudTrail Lambda

Now that we have an event configured to force a remediation of a broken CloudTrail configuration, let’s force it to execute.

  1. In AWS CloudTrail, go to the trail we created in the first lab and remove the CloudWatch Logs Configuration by clicking on the trail, then click on Edit next to CloudWatch Logs. Finally, deselect the Enabled box and save your changes.

  2. Navigate to our AWS Config rule for CloudTrail, and re-evaluate the rule. Refresh the screen after a minute and ensure it comes up as non-compliant.

  3. Go Back to CloudTrail, Did the CloudWatch Log configuration return? Did you get an e-mail?

What did we learn?
  • How to use CloudWatch Events to automatically trigger Lambda functions and automatically remediate non-compliant resources
  • Multiple ways to automate and remediate resources that drift within AWS

Ensure the CloudWatch Agent is Installed on Instances

Now we will create a State Manager job that will run on a schedule to make sure the latest version of the Amazon CloudWatch agent is installed on our instance. System Manager State Manager is a secure and scalable configuration management service that ensures your Amazon EC2 and hybrid infrastructure is in an intended or consistent state, which you define.

  1. Go to the AWS System Manager Console, and on the left side under Actions click on State Manager.
  2. Click on Create Association
  3. Enter CloudWatchAgentInstall as the association name
  4. Under Document, click the radio button next to AWS-ConfigureAWSPackage command
  5. Under Parameters
    • Action: Install
    • Installation type: In-place update
    • Name: AmazonCloudWatchAgent
  6. Under Targets, select Choose instances manually, and then check to select the EC2 instance that you craeted earlier in the lab.
    • If this is not a shared or pre-existing AWS account then there is likely only a single instance in the list
  7. Under Specify schedule
    • Select Radio Button next to CRON schedule builder
    • Enter Every Day at 22:30
  8. Click on Create Association

AWS Config State Manager Association

This association will run every day at 10:30 PM and make sure the latest version of the CloudWatch Agent is installed. We can then run another association to pull down the configuration from the Parameter Store

What did we learn?
  • How to use AWS Systems Manager State Manager to mensure that the CloudWatch Agent is installed and up to date
  • We can use AWS Systems Manager State Manager to ensure a certain state of our EC2 instances

Observe Configuration Timeline and Compliance Timeline

We can view the timeline of compliance and configuration changes for resources inside of your AWS environment from directly within the AWS Config console. To do so, follow these steps:

  1. Go to AWS Config, and click on Resources
  2. Select EC2 Instance, and then click Lookup
  3. Click into the instance we deployed earlier in this lab, and then click on Configuration timeline:

  1. Observe the timeline, and the changes that occurred - most especially the Changes and Relationships:

  1. Then click on Managed instance information in the right-hand corner to see the integration data between AWS Config and AWS Systems Manager.

AWS Config maintains an internal database of relationships between resources in your environment. This lab only shows the basic interactions you can have with the Config resource database.

Observe the CloudTrail Compliance Timeline

  1. Go to AWS Config, and then click on Resources, or if you are already within an instance view, click on Compliance timeline

  1. Select EC2 Instance, and then click Lookup

  2. Click into an EC2 instance we deployed in this lab, then click on Compliance timeline

  1. Observe the timeline, and the compliance changes that occurred. Note that every compliance state change as details that are navigable from directly in the console, informing you as to when exactly a resource became non-compliant.

CloudTrail and CloudWatch Logs Insights

Our final step is to use Amazon Cloudwatch Insights to investigate the volume of activity created by our work in the lab today. CloudWatch Insights is a powerful query engine that an analyze activity across CloudWatch log groups, and our lab has generated a great deal of activity that we can invetigate.

  1. Navigate to the Amazon CloudWatch Logs console, and then from the left-side click on Insights
  2. Choose the Log Group you want to work with from the drop down, and then select management-tools-week
  3. Choose to define a relative time for the past three hours
  4. On the right, click on Queries, then expand Sample queries, CloudTrail, Number of log entries by service, event, and region, and finally click Apply

CW Insights ManagementToolsWeek/CloudTrail

  1. Click Run query

To limit your search to only activity related to AWS Config, modify your query to reflect this:

fields @message
| filter eventSource = ''