Customers often need an easy way to monitor the URLs, API endpoints, and critical GUI workflows of their web applications in a secure fashion. Monitoring helps keep the service available by detecting performance bottlenecks and operational issues as soon as they arise. Customers also want to be alerted when availability and latency issues occur so that they can quickly respond and address them.

In this blog post, I explain how to create, deploy, and monitor synthetic monitoring solution using Amazon CloudWatch Synthetics. Instead of exposing the application credentials in a script, I show you how to integrate the solution with AWS Secrets Manager to manage them in a secure manner.

Overview

Amazon CloudWatch Synthetics is a fully managed synthetic monitoring service that allows developers and DevOps engineers to measure availability of their application endpoints and URLs. It uses configurable scripts called canaries that run 24×7. Canaries alert you as soon as an API, URL, or endpoint does not work as expected, as defined by the canary script. CloudWatch Synthetics canaries can be customized to check for availability, latency, transactions, broken or dead links, and step-by-step task completions. They can also check for page load errors, load latencies for UI assets, complex wizard flows, or checkout flows in your applications.

AWS Secrets Manager helps you protect secrets required to access your applications, services, and IT resources. The service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. AWS IAM users and applications retrieve secrets with a call to Secrets Manager API operations, reducing the need to hardcode sensitive information in plaintext.

Solution architecture

The solution for monitoring end-to-end customer user flows in a secure fashion requires the creation of:

  • An Amazon CloudWatch Synthetics canary for a critical workflow of a sample website.
  • Alarms in CloudWatch to get notifications and to integrate the canary with AWS Secrets Manager to retrieve credentials.

You can easily deploy this solution by using the AWS CloudFormation script given in the later section.

I use a pre-provisioned version of the bookstore demo application to demonstrate a GUI-based workflow using Amazon CloudWatch Synthetics. To use this application, you must first create a user name and password. These credentials are stored in Secrets Manager and are retrieved using the canary scripts.

Implementing the solution

To implement this solution, you must:

  • Create an Amazon CloudWatch Synthetics canary.
  • Create an AWS Secrets Manager secret.
  • Attach Secrets Manager IAM policy to CloudWatch Synthetics role.
  • Modify the canary script to retrieve secrets from Secrets Manager.
  • Deploy the same solution using AWS CloudFormation template (Optional)

Prerequisites

Creating a canary in Amazon CloudWatch Synthetics

 To begin, create an Amazon CloudWatch Synthetics canary to monitor your website. 

  1. Sign in to the AWS Management Console.
  2. In the Search box, enter CloudWatch.
  3. In the CloudWatch console, under Synthetics, choose Canaries.
  4. Choose Create canary.
The Canaries section includes columns for name, last run, success percentage, created, state, and alarms. It also includes a Create canary button.

Figure 1: Create canary button

  1. On the Create canary page, choose Use a blueprint and GUI workflow builder.
Alt Text - The create canary specifies options to create a canary, which includes using a blueprint, inline editor, and import from Amazon S3. It also includes different blueprints to select from when you choose to use a blueprint.

Figure 2: Use a blueprint and GUI workflow builder selected in the CloudWatch console

  1. In Canary builder, enter a name for the canary (aws-book-store-demo).
  2. Enter the application or endpoint URL for your website. For this blog post, I have given the URL of the AWS Book store demo application.

The canary builder section specifies two input boxes where the IAM user can enter the name of the canary and the application URL for monitoring.

Figure 3: Canary builder section

  1. In the Workflow builder section, use the Action, Selector, and Text boxes to specify the actions you want the canary to take on your website. Selectors are HTML elements behind click and type actions on your website (for example, href, id, button, div).

In the preprovisioned version of the application, here is the end-to-end workflow:

Create a user name and password, and sign in to the application. Navigate to the home page and search for a book named database. Navigate to the results page, and on the first result, choose the Add to cart button. Navigate to the checkout page and choose Log out to complete the workflow.

If you click the HTML element on the webpage, you can get the name of the selector from browser developer tools for use in the following script. You can also use CloudWatch Synthetics Recorder to generate user flow scripts for canaries.

These actions are for the flow performed on the sample application.

    • In Action, choose Click with navigation. In Selector, enter [href='/login']. Choose Add action.
    • In Action, choose Input text. In Selector, enter [id='email']. In Text, enter email. Choose Add action. You will replace the user name as an AWS Secrets Manager variable after the script is generated.
    • In Action, choose Input text. In Selector, enter [id=password]. In Text, enter password. Choose Add action. You will replace the password as an AWS Secrets Manager variable after the script is generated.
    • In Action, choose Click with navigation. In Selector, enter button[class='btn btn-lg btn-default btn-block']. Choose Add action.
    • In Action, choose Verify Selector. In Selector, enter [id='txtSearch']. Choose Add action.
    • In Action, choose Input text. In Selector, enter [id='txtSearch']. In Text, enter database. Choose Add action.
    • In Action, choose Click with navigation. In Selector, enter button[class='btn btn-orange no-radius']. Choose Add action.
    • In Action, choose Verify Selector. In Selector, enter div[class='container-category']. Choose Add action.
    • In Action, choose Click with navigation. In Selector, enter button[class='btn btn-black btn-black-center']. Choose Add action.
    • In Action, choose Verify Selector. In Selector, enter div[class='media']. Choose Add action.
    • In Action, choose Click with navigation. In Selector, enter [href='#'].
The workflow builder section specifies the page-specific actions the users can add to build a navigation workflow for their application.

Figure 4: Workflow builder section

  1. In Screenshots section, select Take Screenshots
  2. In Script editor section, keep the Runtime version as syn-nodejs-2.1
  3. In the Schedule section, choose Run continuously and keep the default frequency, in minutes, at which your canary should run. Do not select the Start immediately after creation check box.
  4. In the Additional configuration section, enter the default timeout value for the canary in minutes and seconds.
The schedule section specifies options for the users to select the schedule of the canary run and additional configuration of the timeout values.

Figure 5: Schedule and Additional configuration sections

  1. In the Data retention section, for Failure data retention and Success data retention, keep the default periods for the canary data.
  2. In the Data Storage section, keep S3 location at the default. CloudWatch Synthetics creates a bucket for the data storage.
The data retention section specifies options for the users to select the data retention schedule and the S3 location where the canary results are stored.

Figure 6: Data retention and Data Storage sections for the S3 bucket

  1. In the Access permissions section, choose Create a new role.
  2. In the CloudWatch alarms section, choose Add new alarm, and then select Duration and SuccessPercent as metrics.
  3. In the Set notifications for this canary section, choose Create a new topic, provide the topic name and Email endpoints that will receive the notification. Choose Create topic.

Note: You will receive a verification email from AWS to confirm the notification subscription. Click the link in the email to verify the subscription.

The CloudWatch alarms and notifications section specifies options for the users to create optional CloudWatch alarms and notifications for the canary.

Figure 7: Create CloudWatch alarms and set notifications for the canary

  1. If your website, or resource is deployed in a VPC, in the VPC settings section, choose your VPC.
  2. In the Tags section, choose Add a new tag button to add tags to canaries to help set permissions, organize, and search for them later.
  3. Keep Active Tracing – Optional section as default.
  4. Choose Create canary.

 Note: This operation might take up to a minute. After the canary is created, a success message appears.

Creating a secret in AWS Secrets Manager

We do not recommend that you store the application user name and password in clear text in the canary script. Follow these steps to store the user name and password in AWS Secrets Manager and retrieve it in the canary script to be used in the workflow action.

  1. Open the AWS Secrets Manager console.
  2. On the Store a new secret section, under Select secret type, choose Other type of secrets.
  3. Under Specify the key/value pairs to be stored in this secret, add rows for username and password, and then enter the application user name and password.
  4. In Select the encryption key, choose DefaultEncryptionKey, and then choose Next.
Storing a new secret in the AWS Secrets Manager specifies the option for the users to select a type of the secret, Secret key/value pair, and the encryption key.

Figure 8: Create secret in the Secrets Manager console.

  1. For the secret name, enter mysecret, enter an optional description, and then choose Next.
  2. Choose Disable automatic rotation, and then choose Next.
  3. Choose Store.

Attaching Secrets Manager access policy to the CloudWatch Synthetics role

  1. Return to the Amazon CloudWatch console, and choose your canary from the list.
  2. On the canary details page, choose the Configuration tab.
  3. Under Executed with role, choose the IAM role.
  4. Choose Add inline policy.
  5. On the JSON tab, enter the following policy. Replace the AWS account ID placeholder with your account ID and Region with AWS Region where is canary is created.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "synthetics", "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:Region:Your AWS Account ID:secret:mysecret*" } ]
}
  1. Choose Review Policy, enter a name for the policy (cw-synthetics-sm-policy), and then choose Create Policy.

Modifying the canary script to retrieve the secrets from AWS Secrets Manager 

  1. To integrate with AWS Secrets Manager, add the following code to the canary script.
  2. On the canary details page, from Actions, choose Edit.
  3. Replace the script under Script editor section with the script given below and Choose Save.
var synthetics = require('Synthetics');
const log = require('SyntheticsLogger');
const AWS = require('aws-sdk');
const secretsManager = new AWS.SecretsManager(); const flowBuilderBlueprint = async function () { // INSERT URL here let url = "https://d2h3ljlsmzojxz.cloudfront.net/"; // Get synthetics configuration let syntheticsConfig = synthetics.getConfiguration(); // Set configuration values syntheticsConfig.setConfig({ screenshotOnStepStart : true, screenshotOnStepSuccess: true, screenshotOnStepFailure: true }); // Create a client for Secrets Manager const getSecrets = async (secretName) => { var params = { SecretId: secretName }; return await secretsManager.getSecretValue(params).promise(); } // Fetch secrets let secrets = await getSecrets("mysecret") let secretsObj = JSON.parse(secrets.SecretString); let page = await synthetics.getPage(); // Navigate to the initial url await synthetics.executeStep('navigateToUrl', async function (timeoutInMillis = 30000) { await page.goto(url, {waitUntil: ['load', 'networkidle0'], timeout: timeoutInMillis}); }); // Execute customer steps await synthetics.executeStep('redirection', async function () { await Promise.all([ page.waitForNavigation({ timeout: 30000 }), await page.click("[href='/login']") ]); }); await synthetics.executeStep('input', async function () { await page.type("[id='email']", secretsObj.username); }); await synthetics.executeStep('input', async function () { await page.type("[id=password]", secretsObj.password); }); await synthetics.executeStep('redirection', async function () { await Promise.all([ page.waitForNavigation({ timeout: 30000 }), await page.click("button[class='btn btn-lg btn-default btn-block']") ]); }); await synthetics.executeStep('verifySelector', async function () { await page.waitForSelector("[id='txtSearch']", { timeout: 30000 }); }); await synthetics.executeStep('input', async function () { await page.type("[id='txtSearch']", "database"); }); await synthetics.executeStep('redirection', async function () { await Promise.all([ page.waitForNavigation({ timeout: 30000 }), await page.click("button[class='btn btn-orange no-radius']") ]); }); await synthetics.executeStep('verifySelector', async function () { await page.waitForSelector("div[class='container-category']", { timeout: 30000 }); }); await synthetics.executeStep('redirection', async function () { await Promise.all([ page.waitForNavigation({ timeout: 30000 }), await page.click("button[class='btn btn-black btn-black-center']") ]); }); await synthetics.executeStep('verifySelector', async function () { await page.waitForSelector("div[class='media']", { timeout: 30000 }); }); await synthetics.executeStep('redirection', async function () { await Promise.all([ page.waitForNavigation({ timeout: 30000 }), await page.click("[href='#']") ]); }); }; exports.handler = async () => { return await flowBuilderBlueprint();
};

Note: This operation might take up to a minute. When it is complete, a success message appears.

  1. Choose the canary you updated and from Actions, choose Start. This starts the canary execution. You should see a successful execution of the script.
Successful creation of canary shows the status, previous canary runs and the list of canaries for the user.

Figure 9: Canaries page

Deploying the canary using AWS CloudFormation script (Optional)

You can deploy the canary you created in the previous section using the following AWS CloudFormation script.

Description: "This template creates an Amazon Cloudwatch Synthetics Canary and required resources"
Parameters: UserName: Description: Username for your application to be stored in the Secret Manager Type: String Password: Description: Password for your application to be stored in the Secret Manager Type: String CanaryName: Type: String Description: Name of your Canary. A name consists of up to 21 lowercase letters, numbers, hyphens or underscores with no spaces. CanarySuccessLowAlarmName: Type: String Description: Name of your CW Alarm for Success Percent. CanaryDurationAlarmName: Type: String Description: Name of your CW Alarm for Duration.
Resources: SecretManagerSecret: Type: "AWS::SecretsManager::Secret" Properties: Description: Secret for the application SecretString: !Join [ "", ["{\"username\":\"",!Ref UserName,"\",\"password\":\"",!Ref Password,"\"}"] ] Name: myappsecret SyntheticsLambdaExecutionRole: Type: "AWS::IAM::Role" Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Sid: "" Effect: "Allow" Principal: Service: "lambda.amazonaws.com" Action: "sts:AssumeRole" SyntheticsLambdaExecutionRolePolicies: Type: "AWS::IAM::Policy" Properties: PolicyName: "SyntheticsLambdaExecutionRolePolicy" PolicyDocument: Version: "2012-10-17" Statement: - Action: - "s3:PutObject" - "s3:GetBucketLocation" Resource: - !Sub "arn:aws:s3:::${ArtifactS3LocationBucket}/*" Effect: "Allow" - Action: - "s3:ListAllMyBuckets" - "xray:PutTraceSegments" Resource: "*" Effect: "Allow" - Action: - "cloudwatch:PutMetricData" Resource: "*" Effect: "Allow" - Action: - "logs:*" Resource: Fn::Sub: "arn:${AWS::Partition}:logs:*:*:*" Effect: "Allow" - Action: - "secretsmanager:GetSecretValue" Resource: !Ref SecretManagerSecret Effect: "Allow" Roles: - !Ref "SyntheticsLambdaExecutionRole" ArtifactS3LocationBucket: Type: "AWS::S3::Bucket" Properties: BucketName: !Join [ '', ['cw-syn-results-', !Ref 'AWS::Region', '-', !Ref 'AWS::AccountId' ] ] BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: AES256 LambdaWaiter: Type: "AWS::Lambda::Function" Properties: Handler: "index.handler" Runtime: "nodejs12.x" Timeout: 900 Role: !GetAtt SyntheticsLambdaExecutionRole.Arn Code: ZipFile: | exports.handler = function(e,r) { var response = require('cfn-response'); var waitTimeSeconds = e.ResourceProperties.WaitSeconds; var waitTime = waitTimeSeconds * 1000; setTimeout(function(){ response.send(e, r, response.SUCCESS, {}, "Waiter-" + waitTimeSeconds + "-Seconds") }, waitTime); }; SyntheticsCanary: Type: AWS::Synthetics::Canary DependsOn: - SecretManagerSecret - SyntheticsLambdaExecutionRole - SyntheticsLambdaExecutionRolePolicies Properties: Name: !Ref CanaryName Code: Handler: "flowBuilderBlueprint.handler" Script: "var synthetics = require('Synthetics');\nconst log = require('SyntheticsLogger');\nconst AWS = require('aws-sdk');\nconst secretsManager = new AWS.SecretsManager();\n\n\nconst flowBuilderBlueprint = async function () {\n // INSERT URL here\nlet url = \"https://d2h3ljlsmzojxz.cloudfront.net/\";\n\n // Get synthetics configuration\n let syntheticsConfig = synthetics.getConfiguration();\n\n // Set configuration values\n syntheticsConfig.setConfig({\n screenshotOnStepStart : true,\n screenshotOnStepSuccess: true,\n screenshotOnStepFailure: true\n });\n \n // Create a client for Secrets Manager\n const getSecrets = async (secretName) => {\n var params = {\n SecretId: secretName\n };\n return await secretsManager.getSecretValue(params).promise();\n }\n\n // Fetch secrets\n let secrets = await getSecrets(\"myappsecret\")\n let secretsObj = JSON.parse(secrets.SecretString);\n\n\n let page = await synthetics.getPage();\n\n // Navigate to the initial url\n await synthetics.executeStep('navigateToUrl', async function (timeoutInMillis = 30000) {\n await page.goto(url, {waitUntil: ['load', 'networkidle0'], timeout: timeoutInMillis});\n });\n\n // Execute customer steps\n await synthetics.executeStep('redirection', async function () {\n await Promise.all([\n page.waitForNavigation({ timeout: 30000 }),\n await page.click(\"[href='/login']\")\n ]);\n });\n await synthetics.executeStep('input', async function () {\n await page.type(\"[id='email']\", secretsObj.username);\n });\n await synthetics.executeStep('input', async function () {\n await page.type(\"[id=password]\", secretsObj.password);\n });\n await synthetics.executeStep('redirection', async function () {\n await Promise.all([\n page.waitForNavigation({ timeout: 30000 }),\n await page.click(\"button[class='btn btn-lg btn-default btn-block']\")\n ]);\n });\n await synthetics.executeStep('verifySelector', async function () {\n await page.waitForSelector(\"[id='txtSearch']\", { timeout: 30000 });\n });\n await synthetics.executeStep('input', async function () {\n await page.type(\"[id='txtSearch']\", \"database\");\n });\n await synthetics.executeStep('redirection', async function () {\n await Promise.all([\n page.waitForNavigation({ timeout: 30000 }),\n await page.click(\"button[class='btn btn-orange no-radius']\")\n ]);\n });\n await synthetics.executeStep('verifySelector', async function () {\n await page.waitForSelector(\"div[class='container-category']\", { timeout: 30000 });\n });\n await synthetics.executeStep('redirection', async function () {\n await Promise.all([\n page.waitForNavigation({ timeout: 30000 }),\n await page.click(\"button[class='btn btn-black btn-black-center']\")\n ]);\n });\n await synthetics.executeStep('verifySelector', async function () {\n await page.waitForSelector(\"div[class='media']\", { timeout: 30000 });\n });\n await synthetics.executeStep('redirection', async function () {\n await Promise.all([\n page.waitForNavigation({ timeout: 30000 }),\n await page.click(\"[href='#']\")\n ]);\n });\n\n \n};\n\nexports.handler = async () => {\n return await flowBuilderBlueprint();\n};" ExecutionRoleArn: Fn::GetAtt: SyntheticsLambdaExecutionRole.Arn ArtifactS3Location: !Join [ '', ['s3://', !Ref ArtifactS3LocationBucket] ] RuntimeVersion: "syn-nodejs-2.1" Schedule: {Expression: 'rate(5 minutes)', DurationInSeconds: '3600'} RunConfig: {TimeoutInSeconds: 60} FailureRetentionPeriod: 30 SuccessRetentionPeriod: 30 StartCanaryAfterCreation: true CanarySuccessLowCWAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: !Ref CanarySuccessLowAlarmName AlarmDescription: The CloudWatch Synthetics Canary that monitors the Application Namespace: CloudWatchSynthetics MetricName: SuccessPercent Dimensions: - Name: CanaryName Value: !Ref CanaryName Statistic: Average ComparisonOperator: LessThanOrEqualToThreshold Threshold: 0 Period: 300 # 5-min EvaluationPeriods: 2 # Alarm if the canary is not running/failing for 10 minutes TreatMissingData: breaching # Catch if the canary is not running CanaryDurationCWAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmName: !Ref CanaryDurationAlarmName AlarmDescription: The CloudWatch Synthetics Canary that monitors the Application Namespace: CloudWatchSynthetics MetricName: Duration Dimensions: - Name: CanaryName Value: !Ref CanaryName Statistic: Average ComparisonOperator: GreaterThanOrEqualToThreshold Threshold: 300000 # 5-min Period: 300 # 5-min EvaluationPeriods: 2 # Alarm if the canary is not running/failing for 10 minutes TreatMissingData: breaching # Catch if the canary is not running WaiterCustomResource: Type: "AWS::CloudFormation::CustomResource" DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: ServiceToken: !GetAtt LambdaWaiter.Arn WaitSeconds: 120 Outputs: CanaryName: Value: !Ref "SyntheticsCanary"

Follow these steps to deploy the canaries using AWS CloudFormation.

  1. Use the preceding code block to save a template as a local file (.yml extension) on your computer.
  2. Open the AWS CloudFormation console.
  3. In the left navigation pane, choose Stacks, and then choose Create stack.
  4. In Specify template, choose Upload a template file, and then choose the AWS CloudFormation template file you saved. Choose Next.
  5. Enter a name for your stack (for example, aws-synthetics-stack).
  6. In Parameters, provide the following, and then choose Next.
    • CanaryDurationAlarmName: The alarm name for the Duration metric(for example, mycanary-durationalarm).
    • CanaryName: The name of your canary (for example, mycanary).
    • CanarySuccessLowAlarmName: The alarm name for the SuccessPercent metric. (for example, mycanary-successlowalarm).
    • Password: The password for your web application.
    • UserName: The user name for your web application. 
  7. Define your tags, and then choose Next. (Optional)
  8. On the last page, in Capabilities, be sure to select the check box, and then choose Create stack.

Cleanup

To avoid charges to your account, remove the resources you created.

  1. In the Amazon CloudWatch console, choose the canaries you created and from Actions, choose Stop.
  2. Choose the canaries again and from Actions, choose Delete.

When you delete a canary, the resources used and created by the canary are not deleted automatically. Also delete the following:

  • In the Amazon CloudWatch console, delete the CloudWatch alarms created for this canary. These alarms have a name of Synthetics-Alarm-MyCanaryName.
  • In the Amazon CloudWatch console, delete the CloudWatch log groups created for the canary. These logs groups have the name aws/lambda/cwsyn-MyCanaryName.
  • In the AWS Lambda console, delete the Lambda function used by this canary. These have the prefix cwsyn-MyCanaryName.
  • In the Amazon S3 console, delete the Amazon S3 objects and buckets created for this canary, such as the canary’s artifact location.
  • In the AWS Identity and Access Management console, delete the IAM roles created for the canary. If they were created in the console, these roles have the name role/service-role/CloudWatchSyntheticsRole-MyCanaryName.
  • In the AWS Secrets Manager console, delete the secrets created for the canary (for example, mysecret). Remember that AWS Secrets Manager enforces a minimum waiting period of 7 days to give you time to update your code. You will not be able to retrieve the secret if it is scheduled for deletion.

If you have created your canary using the AWS CloudFormation script, following steps can help you delete the resources:

  • Navigate to Amazon S3 console, delete the Amazon S3 objects in the canary’s artifact bucket. These have a prefix cw-syn-results.
  • Navigate to the AWS CloudFormation console, select the stack you created (aws-synthetics-stack) and click Delete from the menu.
  • Click Delete stack from the confirmation pop-up.

Conclusion

In this post, I’ve shown how you can use Amazon CloudWatch Synthetics and AWS Secrets Manager to automatically monitor the end-to-end workflows of your web applications in a secure manner. I’ve also shown how alarms and notifications can be set to help you troubleshoot availability and latency issues in your application.

You can read more about Amazon CloudWatch Synthetics in the user guide.

About the Author

Author Venugopalan VasudevanVenugopalan Vasudevan is a Senior Technical Account Manager with AWS based in Denver, Colorado. Venu works with AWS customers to solve architectural, operational, and cost optimization challenges and help them build on AWS. In his spare time, he enjoys playing with his two kids.