Offering products to your consumers in AWS is a great way to accelerate adoption, and offering these products through AWS Service Catalog helps to simplify and streamline the process. This blog post describes how you can offer multiple consumers access to your backend products in AWS by using some simple AWS tools and services.

In this case, the backend product uploads newly created or modified objects from an AWS CodeCommit repository to a repository-specific path in an Amazon S3 bucket via some logic in an AWS Lambda function. This method works equally well with any other backend AWS service and is particularly useful for CI/CD or machine learning pipelines in which some logic is required before the pipeline processes the files. In a recent project, I used this method to push machine learning models to dynamically created Amazon EMR clusters.

Overview

The architecture behind the customer-facing portion of this solution is relatively simple, using only three AWS services. As discussed in the summary, the backend architecture uses a single Lambda function to push objects to Amazon S3. In reality, this could be a much larger and more complex solution.

Architecture diagram showing that we only need three AWS Services for this example

Getting started

This example deploys all components of this infrastructure as code using AWS CloudFormation. AWS CloudFormation templates are deployed using AWS CLI. You can deploy them using the AWS console if you prefer, but that is not covered in this blog post.

Prerequisites

This post assumes that you have an AWS account in place with permissions to allow the following:

  • Access to create AWS Lambda functions
  • Access to create AWS CodeCommit repositories and push to them
  • Access to create AWS Service Catalog products
  • Access to create and subscribe to Amazon SNS topics
  • AWS CLI Installed with the above access to your AWS account
  • Amazon S3 bucket created

You should download the AWS CloudFormation templates for this project, unzip them, and store them in a local folder.

Deploying the backend service

In the source code for this blog post, find an AWS CloudFormation template called backend-function.yml. This is the backend service with which you interact. When you create your repository through AWS Service Catalog, you specify this backend service as an input, which allows your single AWS Service Catalog product to serve many different backend products.

  1. Download the backend AWS CloudFormation templates as discussed in the Prerequisites section, unzip them, and place them in a folder on your local computer
  2. Navigate to that folder and run the following AWS CLI command. In this command, you assume that you act on commit to the master branch of your repository. If this is not the case, change the codeCommitBranch key to the branch on which you are acting. You should also replace the value <myS3Bucket> with the correct name for your Amazon S3 bucket.
    aws cloudformation create-stack --stack-name myBackendFunction --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM --template-body file://backend-function.yml --parameters ParameterKey=codeCommitBranch,ParameterValue=master ParameterKey=s3BucketName,ParameterValue=<myS3Bucket>
    This returns a stack ID such as the following:
    {
    "StackId": "arn:aws:cloudformation:eu-west-1:737661087350:stack/myBackendFunction/c0d04af0-f98a-11e9-8f65-06c34fd08df4"
    }
  3. You can check on the progress of the AWS CloudFormation stack creation by running the following command and looking at the StackStatus.
    aws cloudformation describe-stacks --stack-name "<StackId from the above command>"
    Once your status is set to CREATE_COMPLETE, you can continue to the next step.
  4.  Looking at the output from the aws cloudformation describe-stacks command, you should also note down the ExportName in the Outputs section. This is the value that you use when provisioning the CodeCommit repositories so that they connect to this specific backend product. In this case, the name is myBackendFunction-BackendLambdaCode.

Deploying the Service Catalog product

In the folder that you downloaded and unzipped the project files into, find an AWS CloudFormation template called service-catalog-product.yml. This is the code that creates the service catalog product for your consumers and contains the CodeCommit repository that they use. It does this by calling another AWS CloudFormation template that you upload to your Amazon S3 bucket.

  1. In the folder into which you downloaded and unzipped the project files, find an AWS CloudFormation template called create-backend-linked-repository.yml. You need to upload this to the Amazon S3 bucket you created. In practice, this is on a secured bucket owned by your infrastructure team, but in this example, place it on the same bucket to which your backend function is writing. Upload it using the following AWS CLI command, where <myS3Bucket> is the name of your Amazon S3 bucket
    aws s3 cp create-backend-linked-repository.yml s3://<myS3Bucket>
  2.  In the folder into which you downloaded and unzipped the project files, find the file named service-catalog-product.yml.
  3. Navigate to the local folder with the files you downloaded and run the following AWS CLI command. You should replace the value <myS3Bucket> with the correct name for your Amazon S3 bucket, and replace the value <permissionArn> with the full ARN of a user, group, or role that needs to be able to deploy the repositories from the AWS Service Catalog.
    aws cloudformation create-stack --stack-name myServicCatalogProduct --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM CAPABILITY_IAM --template-body file://service-catalog-product.yml --parameters ParameterKey=s3BucketName,ParameterValue=<myS3Bucket> ParameterKey=permissionsArn,ParameterValue=< permissionsArn >
    This returns a stack ID such as the following:
    {
    "StackId": "arn:aws:cloudformation:eu-west-1:737661087350:stack/myServicCatalogProduct/dcb48f80-f988-11e9-8199-0637bdb794d0"
    }
  4.  You can check on the progress of the AWS CloudFormation stack creation by running the following command and looking at the StackStatus:
    aws cloudformation describe-stacks --stack-name "<StackId from the above command>"

Once your status is set to CREATE_COMPLETE, you can continue to the next step.

Deploying the AWS CodeCommit repository as a user

Now that you have deployed the infrastructure around this AWS Service Catalog product, you can deploy the actual repository just as a user would. You do this from the AWS Service Catalog page in the AWS console.

  1. Open the AWS Service Catalog page and navigate to the product lists. You should see the product you just created, called CodeCommit Repository for Demo. Choose the product name, and then choose Launch Product.
  2. Give the product a name and choose Next.
  3. Enter the details into the Parameters page. You can leave the default values in there for this example or change the values to something more meaningful. The parameter for backendFunction should be the name of the backend function. This is the ExportName that you noted down in Step 4 in the Deploying the backend service section of this blog (in this case it is myBackendFunction-BackendLambdaCode).
  4. Enter any tags that you want to use and then choose Next.
  5. Leave the checkbox unselected in the Notifications section and choose Next.
  6.  Choose Launch to create your new repository.

Uploading content to the AWS CodeCommit Repository

Note that, in the AWS CodeCommit console, you have created a new repository. You can now choose the Clone URL links (either HTTPS or SSH) and connect from your favorite Git client, as shown in the following screenshot.

View of the CodeCommit Repository that was created in the previous step

If you prefer, you can also use the AWS CodeCommit user interface to add and update your files, as shown in the following screenshot.

Adding files directly to the Repository using the AWS CodeCommit UI

Once you commit to the master branch, you can see your files in the Amazon S3 bucket you referenced for this project, which validates that your integration has worked.

Cleaning up your environment

There are three steps to cleaning up your environment after deploying this infrastructure. You must first remove any AWS CodeCommit repositories that you provisioned using AWS Service Catalog, then remove the infrastructure AWS CloudFormation Templates that you deployed and finally you should remove any data that you pushed into your AWS CodeCommit repository from Amazon S3.

Since the end user created the AWS CodeCommit Repository via AWS Service Catalog, we will get them to remove these repositories in the same way.

  1. Open the AWS Service Catalog page and navigate to the Provisioned product list. You should see the repository that you created earlier. Hit the three dots to the left of the product and select Terminate provisioned product.
  2. Click Terminate in the warning window that appears.
  3. After a few minutes hit the refresh button and you will see that this provisioned product disappears.

Snip showing how to terminate an AWS Service Catalog provisioned products

Now that we have cleaned up our repositories, we need to remove the AWS CloudFormation stacks that contain all of the logic. Since we deployed these using the AWS CLI, we will remove them in the same way.

  1. You should first remove the Service Catalog stack by running the command:
    aws cloudformation delete-stack --stack-name myServicCatalogProduct
  2. You can check on the progress of the AWS CloudFormation stack deletion by running the following command and looking at the StackStatus:
    aws cloudformation list-stacks
    When this stack shows a StackStatus of DELETE_COMPLETE then it has been successfully removed and you can move onto the next step.
  3. Next you need to remove the backend stack. You can do this by running the following command:
    aws cloudformation delete-stack --stack-name myBackendFunction
  4. You can check on the progress of the AWS CloudFormation stack deletion by running the following command and looking at the StackStatus:
    aws cloudformation list-stacks
    When this stack shows a StackStatus of DELETE_COMPLETE then it has been successfully removed and you can move onto the next step.

Finally you should remove any unwanted test data from the Amazon S3 bucket that you chose as a target for our repository. All objects will be in a folder with the same name as the repository and this whole folder can now be removed. Please ensure that any data being removed is no longer required before deleting.

Conclusion

In this blog post, you used AWS CloudFormation and AWS CLI to deploy an AWS Service Catalog product and associated a backend Lambda function to move files from a CodeCommit repository to an Amazon S3 bucket. As previously discussed, this is a simple use case for what you can do using this type of infrastructure. By changing the Lambda function to match your requirements, you can use the same infrastructure for practically anything.

 

6OaiGe06htI

from AWS DevOps Blog: https://ift.tt/34XvLRz

Categories: DevOps