Many of you have experience using AWS CloudFormation to automate your application deployments. As you probably know, the service supports around 600 types of resources. When you optimize your templates, you might have discovered that each of those resource types encapsulates native AWS SDK API calls to create or update each resource’s state or configuration. You might also have discovered that with more than 200 AWS services, it takes some time for API updates to be enabled for some resources in AWS CloudFormation.

In this blog post, we focus on helping you use easy, and safe custom resources in AWS CloudFormation for cases when you need those API updates to build reusable templates. Specifically, we show you how to get any attribute from any AWS CloudFormation resource to use in any of the !Ref and !GetAtt intrinsic function calls. We believe this is a great way for you to learn resource customization options while addressing the more than 50 coverage gaps identified in our public roadmap with a safe, easy, reversible workaround. There are four distinct ways to do it, with options to fit your current skills, scenarios, and process maturity levels.

In this two-part series, we describe how to build custom resources using cfn-responsecrhelpermacros, and resource types. This post covers cfn-response and crhelper.

Prerequisites

We use YAML templates and AWS Lambda-backed custom resources written in Python. In some cases, you can use other languages. We assume that you’re familiar with AWS CloudFormation templates, AWS Lambda, and Python.

Option 1: Inline Lambda backed custom resource using cfn-response

Time to read15 minutes
Time to complete~ 20 minutes
Learning levelIntermediate (200)
AWS ServicesAWS CloudFormation
Amazon EC2
Software ToolsAWS CLI
Linux, MacOS, or Windows subsystem for Linux

Of the four options we cover, this one is the quickest from start to finish. You should be able to go from creation to cleanup in 20 minutes or so. This option lets you write your quick (single command) logic embedded in a template. You don’t need to worry about external files or Amazon S3 dependencies. Both template developers and end users get to find what the custom resource does at any given point in time, which makes the template easier to maintain.

The following template addresses the request in GitHub issue 157 to retrieve the Amazon EC2 security group name. Although AWS CloudFormation doesn’t get AWS::EC2::SecurityGroup name by default, you can get it with one additional SDK call without disrupting resource creation or updates.

cfn-ec2-custom-resource.yml

AWSTemplateFormatVersion: '2010-09-09'
Parameters: vpcID: Type: AWS::EC2::VPC::Id Description: Enter VPC Id Resources: CfnEC2SecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: CFN2 Security Group Description VpcId: !Ref vpcID LambdaBasicExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: CustomLambdaEC2DescribePolicy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: arn:aws:logs:*:*:* - Effect: Allow Action: - ec2:DescribeSecurityGroups Resource: '*' CustomSGResource: Type: AWS::CloudFormation::CustomResource Properties: ServiceToken: !GetAtt 'CustomFunction.Arn' ResourceRef: !Ref 'CfnEC2SecurityGroup' CustomFunction: Type: AWS::Lambda::Function Properties: Handler: index.lambda_handler Description: "Retrieves EC2 Security group name" Timeout: 30 Role: !GetAtt 'LambdaBasicExecutionRole.Arn' Runtime: python3.7 Code: ZipFile: | import json import logging import cfnresponse import boto3 logger = logging.getLogger() logger.setLevel(logging.INFO) def lambda_handler(event, context): logger.info('got event {}'.format(event)) try: responseData = {} if event['RequestType'] == 'Delete': logger.info('Incoming RequestType: Delete operation') cfnresponse.send(event, context, cfnresponse.SUCCESS, {}) if event['RequestType'] in ["Create", "Update"]: # 1. retrieve resource reference ID or Name ResourceRef=event['ResourceProperties']['ResourceRef'] # 2. retrieve boto3 client client = boto3.client('ec2') # 3. Invoke describe/retrieve function using ResourceRef response = client.describe_security_groups(GroupIds=[ResourceRef]) # 4. Parse and return required attributes responseData = {} responseData['SecurityGroup-Name']= response.get('SecurityGroups')[0].get('GroupName') logger.info('Retrieved SecurityGroup-Name!') cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData) else: logger.info('Unexpected RequestType!') cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData) except Exception as err: logger.error(err) responseData = {"Data": str(err)} cfnresponse.send(event,context,cfnresponse.FAILED,responseData) return Outputs: SecurityGroupID: Description: Security Group ID Value: !Ref CfnEC2SecurityGroup SecurityGroupName: Description: Security Group Name from the custom resource Value: !GetAtt 'CustomSGResource.SecurityGroup-Name'

What the template does

To make this code reusable, we use Parameters to allow any VPC to be used. In non-EC2 cases, you can pass other parameters.

In the required Resources section, we first create the EC2 security group for the VPC. Next, we handle security controls by creating an AWS IAM role and a policy. This allows the Lambda-backed custom resource to allow onlyec2:DescribeSecurityGroupsAPI actions and to log data to Amazon CloudWatch.

Now for the heavy lifting, we define an inline, Lambda-backed custom resource. This is where you put your custom logic to get the missing EC2 security group name attribute. This function must be designed to account for CREATE, UPDATE, and DELETE invocations, along with a request payload object. As you find in the code, we only have to explicitly implement CREATE or UPDATE invocations. We run our logic by using the boto3 client Python library to directly invoke the describe_security_groups API action. This retrieves the EC2 security group name and returns it by using the cfn-response callback utility module. This module helps Lambda functions communicate with AWS CloudFormation and with the response object it expects. It uses the presigned Amazon S3 ResponseURL that AWS CloudFormation automatically builds in the request object. The Lambda function code contains inline comments for you (and future users of your utility) if you need to change it over time.

Our last resource is where we declare the function as a custom resource using the AWS::CloudFormation::CustomResource naming convention and we must reference the function’s ARN using theServiceTokenproperty. Because we want to get the EC2 security group name, we pass the EC2 security group ID we already have as a parameter. AWS CloudFormation invokes the custom resource only after it has been created, so no additionalDependsOnproperty is required. After the custom resource is created, AWS CloudFormation invokes the function asynchronously. As soon as the function sends the responseData back using the cfn-response module, AWS CloudFormation processes the response and marks the custom resource operation successful if it receives the missing attribute. Otherwise, we want the operation to properly fail.

The Outputs section of the template gives us a way to display the security group name and ID we retrieved with our custom code. We can also use these new acquired attributes as outputs or exports to orchestrate more complex sequences across stacks with nested stacks, cross stack references, and other features.

Now that you understand what we are doing, give it a try!

  1. Copy and paste the template code into an empty file namedcfn-ec2-custom-resource.yml
  2. Run cfn-lint and ensure that there are no errors or warnings. You might want to runcfn-lint –uto ensure you have the latest AWS CloudFormation resource specifications.
  3. Deploy the resources using the deploy AWS CLI command.
  4. To view stack output or events, use the describe-stacks AWS CLI command.

Here are sample commands. It should take you less than two minutes to run them from start to finish.

#copy-paste template code to a file named cfn-ec2-custom-resource.yml and save it!
vim cfn-ec2-custom-resource.yml #validate template against latest AWS CloudFormation Resource specification
cfn-lint -t cfn-ec2-custom-resource.yml #change vpcID parameterValue below
aws cloudformation deploy --stack-name ec2securitygroupstack --capabilities CAPABILITY_IAM --template-file cfn-ec2-custom-resource.yml --parameter-overrides vpcID=CHANGEME #validate stack output aws cloudformation describe-stacks --stack-name ec2securitygroupstack --query "Stacks[0].Outputs" [ { "OutputKey": "SecurityGroupName", "OutputValue": "ec2securitygroupstack-CfnEC2SecurityGroup-182M368T50HSL", "Description": "Security Group Name from the custom resource" }, { "OutputKey": "SecurityGroupID", "OutputValue": "sg-08d735ebb44661108", "Description": "Security Group ID" }
]

We retrieved an additional attribute safely using a simple custom resource! It wasn’t that hard, was it? You essentially used one SDK call to address the attribute gap. You only need to know some basic Python, the boto3 module, and the cfn-response module. TheresponseDatafrom our function call contains the attribute we want, which we now can access like any other built-in AWS CloudFormation resource! We did it in a safe way, using proper logging, exception handling, and with limited read-only permissions. If necessary, you can copy and paste the inline AWS Lambda function code to other templates whenever we need this attribute.

The cfn-response module is available only when you use theZipFileproperty to write your source code directly inline. Due to the 4096 character limit restricting inline function code, cfn-response is suited for simple and small implementations. The template becomes less readable to users and administrators if they are not familiar with Python code. It also stops some linters from highlighting warnings or errors while developing the template, because they can parse a file as YAML or Python, but not both simultaneously.

Cleanup

Use the following command to delete the stack:

aws cloudformation delete-stack --stack-name ec2securitygroupstack

Option 2: Custom resource using AWS Lambda function and crhelper

Time to read15 minutes
Time to complete~ 60 minutes
Learning levelIntermediate (200)
AWS ServicesAWS CloudFormation
AWS Directory Service for Microsoft Active Directory
Amazon FSx for Windows File Server
AWS Secrets Manager
AWS Lambda
Software ToolsAWS CLI
Linux, MacOS, or Windows subsystem for Linux

Now that you have tested option 1, you can use it to deploy a similar template to different workloads or environments in the same account. The cfn-response module only works with inline function code so that you don’t have to build and package it along with the Lambda function. However, you’ll end up duplicating the same custom resource code across templates. Wouldn’t it be nice to have a better option to write this code once and reuse it? That’s why we cover the custom resource helper (crhelper) next.

The following template addresses the request in GitHub issue 446. As of this writing, there’s no way to fetch the DNS name of the AWS::FSx:Filesystem in the current implementation of the AWS CloudFormation resource. We will retrieve the Amazon FSx file system ID as we create the file system using AWS CloudFormation. We will retrieve the DNS name using!GetAttintrinsic function with a custom resource again, but this time we use crhelper. Because AWS::DirectoryService::MicrosoftAD takes 25 minutes to make and AWS::FSx:Filesystem takes another 25 minutes, you should plan for about 50-60 minutes to try out this example.

Written in Python, crhelper simplifies custom resource development by exposing prebuilt decorators for the CREATE, UPDATE, and DELETE requests. This framework includes detailed logging, with options like toggling log levels and JSON formatting, which helps you parse events with any logging tool. It also helps capture all exceptions to track meaningful errors in AWS CloudFormation events. This is done by wrapping the exception trace to a common format like “Failed to create resource: <Reason>” and truncating the errors for better readability. In addition, crhelper is designed to provide a response callback to AWS CloudFormation with a built-in retry mechanism. This retry handling is important, because an uncaught exception or other failure to respond to AWS CloudFormation can make your stack timeout before a rollback is started. For more information about crhelper features and benefits, check the GitHub repository.

We start by creating a file named cfn-fsx-custom-resource.yml.

AWSTemplateFormatVersion: '2010-09-09' Parameters: vpcID: Type: AWS::EC2::VPC::Id Description: Enter VPC Id PrivateSubnet1ID: Type: AWS::EC2::Subnet::Id Description: Enter PrivateSubnet1ID PrivateSubnet2ID: Type: AWS::EC2::Subnet::Id Description: Enter PrivateSubnet2ID Resources: MyADCredentialName: Type: AWS::SecretsManager::Secret Properties: Description: 'This is my AD secret' GenerateSecretString: SecretStringTemplate: '{"username": "customchangeme"}' GenerateStringKey: 'password' PasswordLength: 16 ExcludeCharacters: '"@/\' MyDirectory: Type: AWS::DirectoryService::MicrosoftAD Properties: Name: "corp1.example.com" Edition: Standard Password: !Join - ':' - - '{{resolve:secretsmanager' - !Ref MyADCredentialName - 'SecretString}}' ShortName: CorpExampleAD VpcSettings: SubnetIds: - Ref: PrivateSubnet1ID - Ref: PrivateSubnet2ID VpcId: Ref: vpcID FSxFileSystem: Type: 'AWS::FSx::FileSystem' Properties: FileSystemType: WINDOWS StorageCapacity: 32 StorageType: 'SSD' SubnetIds: - Ref: PrivateSubnet1ID - Ref: PrivateSubnet2ID WindowsConfiguration: ActiveDirectoryId: !Ref MyDirectory ThroughputCapacity: 8 DeploymentType: MULTI_AZ_1 PreferredSubnetId: !Ref PrivateSubnet1ID CustomFSxResource: Type: AWS::CloudFormation::CustomResource Properties: ServiceToken: !Join ["", ["arn:aws:lambda:",{Ref: "AWS::Region"}, ":", {Ref: "AWS::AccountId"}, ":function:crhelper-fsx-resource" ]] ResourceRef: !Ref FSxFileSystem Outputs: FileSystemID: Description: File System ID Value: !Ref FSxFileSystem DNSName: Description: FSx DNSName from the custom resource. Value: !GetAtt 'CustomFSxResource.DNSName'

What the template does

You might have noticed that this template looks a lot like the previous one.

We start withParametersto capture input values like VPC ID and two private subnet IDs, which are required for this use case.

In theResourcessection, we show you how you can use AWS Secrets Manager to store the managed directory’s password and dynamically reference it within the template. This secures your code by avoiding embedded plaintext passwords. We then create the managed directory and a file system for the directory to use. We also create the custom resource usingAWS::CloudFormation::CustomResource. Custom resources require a mandatoryServiceToken. Because we want to retrieve the file system’s DNS name, we pass the file system ID as an additional parameter. AWS CloudFormation follows the workflow defined earlier to invoke the Lambda function asynchronously at creation time and waits for the response from the invocation, which contains the DNS name.

We get to access the DNS name and file system ID and print them in theOutputssection using!GetAttand!Refintrinsic functions, as we did previously.

Aren’t we missing something here? Yes, the Lambda-backed custom resource! One of the advantages of using crhelper is that it helps you decouple the custom resource code from the template. In the next three steps, we show you how to create a standalone Lambda-backed custom resource to get the missing DNS name.

As Lambda users know, you need to create an execution role for the function to assume, so it can access AWS services and resources. Although you could create both the role and the function in a single template, we are explicitly doing it separately, since we want to show how to reuse the same function in any other case that requires the DNS name for other applications using Amazon FSx. You must create the JSON file before executing the IAM create-role CLI command.

LambdaAssumeRole.json

{ "Version": "2012-10-17", "Statement": { "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole" } }
aws iam create-role --role-name CustomResourceLambdaRole --assume-role-policy-document file://LambdaAssumeRole.json

Next, create an inline policy for the role to restrict the function to allow onlyfsx:Describe*API actions and log data into CloudWatch. Like the execution role, first create the JSON file before executing the IAM put-role-policy CLI command.

CustomLambdaFSxReadPolicy.json

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:*:*:*" ] }, { "Effect": "Allow", "Action": [ "fsx:Describe*" ], "Resource": [ "*" ] } ]
}
aws iam put-role-policy --role-name  CustomResourceLambdaRole --policy-name CustomLambdaFSXDescribePolicy --policy-document file://CustomLambdaFSxReadPolicy.json

For the permissions in theActionssection, we choseDescribe*so the policy can be reused for other attributes in the future. You can be more restrictive. As you add other custom resources, you might want to consider how to administer multiple utility policies in the long term. Perhaps put them all in a single AWS CloudFormation template?

Now, it’s time to install the crhelper Python module in an empty directory. Then, we create a Lambda function in the same directory. You only have to install the crhelper module once on your machine. It can be repackaged and reused with any future function.

mkdir lambda_cr_function cd lambda_cr_function #install crhelper in current directory using --target flag
pip install --target . crhelper #copy-paste function code below to a file named lambda_function.py and save it!
vim lambda_function.py

lambda_function.py: 

from __future__ import print_function
from crhelper import CfnResource
import boto3
import logging
logger = logging.getLogger()
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL')
@helper.create
@helper.update
def create(event, context): # 1. retrieve resource reference ID or Name ResourceRef = event['ResourceProperties']['ResourceRef'] # 2. retrieve boto3 client client = boto3.client('fsx') # 3. Invoke describe/retrieve function using ResourceRef response = client.describe_file_systems(FileSystemIds=[ResourceRef]) # 4. Parse and return required attributes helper.Data['DNSName'] = response.get('FileSystems')[0].get('DNSName') logger.info('Retrieved DNSName!')
@helper.delete
def no_op(_, __): pass
def handler(event, context): helper(event, context)

Let’s review the function code. We define the logic to retrieve the DNS name using boto3. Because we have to retrieve the DNS name, we only require CREATE and UPDATE invocation handlers, so we focus on thecreateandupdatedecorators first. Did you notice we are writing a lot less code? We only need one SDK call (describe_file_systems) to retrieve the DNS name. Finally, we pass it back using the CfnResource helper object. Because we just want to retrieve attributes, we use the delete decorator only as a pass-through when it receives a DELETE invocation. When invoked multiple times with the same file system ID, this function returns the same DNS name, making it an idempotent function. We want the function to be idempotent to make sure that retries, updates, or rollbacks don’t create duplicate resources or leave them orphaned.

Now, zip it up and deploy this Lambda function named crhelper-fsx-resource. By keeping the function separate, your peers can reuse it wherever they need it.

zip -r ../lambda_cr_function.zip ./
#Replace 123456789012 with your own account
aws lambda create-function --function-name crhelper-fsx-resource --handler lambda_function.handler --timeout 10 --zip-file fileb://../lambda_cr_function.zip --runtime python3.8 --role "arn:aws:iam::123456789012:role/CustomResourceLambdaRole"

Finally, we create the template and run the same commands we did before. FSx for Windows File Server is not available in all AWS Regions. For a current list of supported Regions, see Service Endpoints and Quotas in the AWS documentation.

#copy-paste template code, save it!
vim cfn-fsx-custom-resource.yml #validate the template against the AWS CloudFormation Resource specification
cfn-lint -t cfn-fsx-custom-resource.yml #change parameters for vpcID, PrivateSubnet1ID and PrivateSubnet2ID
aws cloudformation deploy --stack-name FSxStack --template-file cfn-fsx-custom-resource.yml --capabilities CAPABILITY_IAM --parameter-overrides vpcID=CHANGEME PrivateSubnet1ID=CHANGEME PrivateSubnet2ID=CHANGEME #validate stack output aws cloudformation describe-stacks --stack-name FSxStack --query "Stacks[0].Outputs" [ { "OutputKey": "FileSystemID", "OutputValue": "fs-045f97d3e7f9f39dc", "Description": "File System ID" }, { "OutputKey": "DNSName", "OutputValue": "amznfsxgtco9yqx.corp1.example.com", "Description": "FSx DNSName from the custom resource." }
]

You can use AWS CloudFormation events or CloudWatch Logs to monitor the stack events. After the stack creation is complete, you should see Amazon FSx DNSName and FileSystemID in theOutputssection from describe-stacks AWS CLI command or from the AWS CloudFormation console.

We made another custom resource! With crhelper, you can focus on your logic, not on the boilerplate code required to integrate with AWS CloudFormation. Now you can write and maintain even fewer lines of Python code and reuse it across many templates in the same account.

The key difference between these two options is that cfn-response just helps with the return response object to AWS CloudFormation, while crhelper provides logging, polling, decorators, exception handling, and timeout trapping. You can let crhelper do most of the heavy lifting and implement the actions you need in the required request types.

Separating the Python code from the template makes for better readability and user experience too. If you have programming skills, you should be aware of all custom resource options and learn how to rigorously test, validate, and keep your custom code updated. Compared to newer options like resource types, crhelper does not help with enabling your custom resources to use drift detection and change sets. It also doesn’t help you capture events from your code, because it is maintained separately from the stack. More about this in part two of this blog post.

Cleanup

As we did earlier in the first option, let’s clean up the stack. Use the following commands to delete the resources created in this example. Apart from theFSxStack, because we created a Lambda function, an AWS IAM role and policy, we need to remove them as well.

aws cloudformation delete-stack --stack-name FSxStack aws iam delete-role-policy --role-name CustomResourceLambdaRole --policy-name CustomLambdaFSXDescribePolicy
aws iam delete-role --role-name CustomResourceLambdaRole aws lambda delete-function --function-name crhelper-fsx-resource

Conclusion

We covered two of four ways to do easy and safe customizations to address gaps related to attributes that are not yet available in AWS CloudFormation. By handling failures gracefully, setting reasonable timeout periods, and implementing request type events properly, custom resources help you build your stacks exactly as you want them.

We hope we’ve shown you that you can customize AWS CloudFormation in a safe and maintainable way. And, if we didn’t quite do that, keep in mind that we have two more options to show you.

Part 2 covers more advanced techniques, including the benefits of doing the same!GetAttoperations with macros. We also dive deep into the latest resource type option, which makes it possible to detect drift with your custom resources.

Further reading

Beyond these!GetAttcases, these options also help you integrate and provision non-AWS resources and perform tasks specific to your organization, or tasks not related to infrastructure directly. Consider a use case where, after AWS CloudFormation creates your Amazon RDS database, you can use one of these options to create a table in that database. For more information, check custom resources in the AWS CloudFormation documentation.

These are just two of the many options you can use to address the over 50 issues tracked in AWS CloudFormation public roadmap. There is so much to discover about these and other customization options. For more information, check cfn-response modulecrhelper framework, and custom resource examples for AWS CloudFormation on GitHub.

About the Author

Gokul Sarangaraju profile imageGokul Sarangaraju is a Senior Technical Account Manager at AWS. He helps customers adopt AWS services and provides guidance in AWS cost and usage optimization. His areas of expertise include delivering solutions using AWS CloudFormation, various other automation techniques. Outside of work, he enjoys playing volleyball and poker – Set, Spike, All-In! You can find him on twitter at @saranggx.
bio2q2020 120 160Luis Colon is a Senior Developer Advocate at AWS specializing in CloudFormation. Over the years he’s been a conference speaker, an agile methodology practitioner, open source advocate, and engineering manager. When he’s not chatting about all things related to infrastructure as code, DevOps, Scrum, and data analytics, he’s golfing or mixing progressive trance and deep house music. You can find him on twitter at @luiscolon1.