AWS customers with Git repositories in Azure DevOps can automatically backup their repositories in the AWS Cloud using an AWS CodeCommit repository as a replica. By configuring an Azure DevOps pipeline, the source and replica repositories can be automatically kept in sync. When updates are pushed to the source repository, the pipeline will be triggered to clone the repository and push it to the replica repository in AWS.

In this post, we show you how to automatically sync a source repository in Azure DevOps to a replica repository in AWS CodeCommit using an Azure DevOps pipeline.

Solution overview

The following diagram shows a high-level architecture of the pipeline.
Solution architecture diagram
To replicate your repository in the AWS Cloud, you perform the following steps which we will cover in this blog post:

  1. Create a repository in CodeCommit.
  2. Create a policy, user, and HTTPS Git credentials in AWS Identity and Access Management (IAM).
  3. Create a pipeline in Azure DevOps.

Prerequisites

Before you get started, make sure you have the following prerequisites set up:

  • An AWS account
  • An Azure DevOps repository

Creating a repository in CodeCommit

You first create a new repository in CodeCommit to use as your replica repository. You need the URL and Amazon Resource Name (ARN) of the replica repository to complete this example pipeline. Follow these steps to create the repository and get the URL and ARN:

  1. Create a CodeCommit repository in the Region of your choice. Choose a name to help you remember that this repository is a replica or backup repository (for example, MyRepoReplica). Important: Do not manually push any changes to this replica repository. It will cause conflicts later when your pipeline pushes changes in the source repository. Treat it as a read-only repository and push all of your development changes to your source repository.
  2. On the AWS CodeCommit console, choose Repositories.
    CodeCommit consol screenshot
  3. Choose your repository and choose View Repository.
    CodeCommit repo screenshot
  4. Choose Clone URL and choose Clone HTTPS. This copies the repository’s URL. Save it by pasting it into a plain-text editor.
    CodeCommit console screenshot
  5. On the navigation pane, under Repositories, choose Settings.
    CodeCommit console screenshot
  6. Copy the value of Repository ARN and save it by pasting it into a plain-text editor.
    CodeCommit repo screenshot

Creating a policy, user, and HTTPS Git credentials in IAM

The pipeline needs permissions and credentials to push commits to your CodeCommit repository. In this example, you create an IAM policy, IAM user, and HTTPS Git credentials for the pipeline to give it access to your repository in AWS. You grant least privilege to the IAM user so the pipeline can only push to your replica repository.

To create the IAM policy, complete the following steps:

  1. On the IAM console, choose Policies.
  2. Choose Create Policy.
  3. Choose JSON.
    IAM console screenshot
  4. Enter a policy that grants permission to push commits to your repository. You can use a policy that’s similar to the following. For the resource element, specify the ARN of your CodeCommit repository:
    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "codecommit:GitPush", "Resource": "arn:aws:codecommit:us-east-1:123456789012:MyRepoReplica" } ]
    }
    

  5. Choose Review policy.
    IAM console screenshot
  6. For Name, enter a name for your policy (for example, CodeCommitMyRepoReplicaGitPush).
  7. Choose Create policy.

For more information, see Creating IAM Policies.

You can now create the IAM user.

  1. On the IAM console, choose Users.
  2. Choose Add user.
  3. Enter a User name (for example, azure-devops-pipeline).
    IAM console screenshot
  4. Select Programmatic access.
    IAM console screenshot
  5. Choose Next: Permissions.
  6. Select Attach existing policies directly and select the IAM policy you created.
    IAM console screenshot
  7. Choose Next: Tags.
  8. Choose Next: Review.
  9. Choose Create user.
  10. When presented with security credentials, choose Close.
    IAM console screenshot
  11. Choose your new user by clicking on the user name link.
  12. Choose Security Credentials.
    IAM console screenshot
  13. Under Access keys, remove the existing access key.
    IAM console screenshot
  14. Under HTTPS Git credentials for AWS CodeCommit, choose Generate credentials.
    IAM console screenshot
  15. Choose Download credentials to save the user name and password.
    IAM console screenshot
  16. Choose Close.

For more information, see Creating an IAM user and Setup for HTTPS users using Git credentials.

Creating a pipeline in Azure DevOps

The pipeline in this post clones a mirror of your source repository and pushes it to your CodeCommit repository. The pipeline requires the URL of your source repository and HTTPS Git credentials to clone it.

To find the URL of your source repository and to generate HTTPS Git credentials, complete the following steps:

  1. Go to the Repos page within Azure DevOps and choose your repository.
  2. Choose Clone.
  3. Choose HTTPS.
  4. Copy and save the URL by pasting it into a plain-text editor.
  5. Choose Generate Git Credentials.
  6. Copy the user name and password and save them by pasting them into a plain-text editor.

Now that you have the URL and HTTPS Git credentials, create a pipeline.

  1. Go to the Pipeline page within Azure DevOps.
  2. Choose Create Pipeline (or New Pipeline).
  3. Choose Azure Repos Git.
    Azure DevOps screenshot
  4. Choose your repository.
    Azure DevOps screenshot
  5. Choose Starter pipeline.
    Azure DevOps screenshot
  6. Enter the following YAML code to replace the default pipeline YAML:
    # Pipeline to automatically mirror
    # an Azure DevOps repository in AWS CodeCommit # Trigger on all branches
    trigger:
    - '*' # Use latest Ubuntu image
    pool: vmImage: 'ubuntu-latest' # Pipeline
    steps:
    - checkout: none
    - script: | # Install urlencode function to encode reserved characters in passwords sudo apt-get install gridsite-clients # Create local mirror of Azure DevOps repository git clone --mirror https://${AZURE_GIT_USERNAME}:$(urlencode ${AZURE_GIT_PASSWORD})@${AZURE_REPO_URL} repo-mirror # Sync AWS CodeCommit repository cd repo-mirror git push --mirror https://${AWS_GIT_USERNAME}:$(urlencode ${AWS_GIT_PASSWORD})@${AWS_REPO_URL} displayName: 'Sync repository with AWS CodeCommit' env: AZURE_REPO_URL: $(AZURE_REPO_URL) AZURE_GIT_USERNAME: $(AZURE_GIT_USERNAME) AZURE_GIT_PASSWORD: $(AZURE_GIT_PASSWORD) AWS_REPO_URL: $(AWS_REPO_URL) AWS_GIT_USERNAME: $(AWS_GIT_USERNAME) AWS_GIT_PASSWORD: $(AWS_GIT_PASSWORD)
    

Add the following variables to your pipeline using the steps below:

NameValueKeep Secret
AZURE_REPO_URLYour Azure DevOps repository URL (do not include https://[email protected])Optional
AZURE_GIT_USERNAMEYour Azure HTTPS Git credentials user nameYES
AZURE_GIT_PASSWORDYour Azure HTTPS Git credentials passwordYES
AWS_REPO_URLYour CodeCommit repository URL (do not include https://)Optional
AWS_GIT_USERNAMEYour AWS HTTPS Git credentials user nameYES
AWS_GIT_PASSWORDYour AWS HTTPS Git credentials passwordYES
  1. Choose Variables.
  2. Choose New Variable.
  3. Enter the variable Name and Value.
  4. Select Keep this value secret when adding any user name or password variable.
    Azure DevOps screenshot
  5. Choose OK.
  6. Repeat for each variable.
    Azure DevOps screenshot
  7. Choose Save.
  8. Choose Save and run.

Verifying the pipeline

When you save the pipeline, it commits the pipeline’s YAML file (azure-pipelines.yml) to the root of your source repository’s primary branch and then runs. You can verify that the pipeline ran successfully by viewing the pipeline job in Azure DevOps pipelines and viewing your replica repository on the CodeCommit console.

  1. Go to the Pipeline page within Azure DevOps and choose your pipeline.
  2. Choose the entry for the latest run.
  3. Under Jobs, choose Job to view the output of your pipeline.
    Azure DevOps screenshot
  4. On the CodeCommit console, choose Repositories.
  5. Choose your repository and choose View Repository.
  6. On the navigation pane, choose Commits.
  7. Verify that the CodeCommit repository contains the latest commit from Azure DevOps.
    CodeCommit console screenshot

The pipeline runs whenever a new commit is pushed to the source repository. All updates are mirrored in the replica CodeCommit repository, including commits, branches, and references.

Cleaning up

When you’ve completed all steps and are finished testing, follow these steps to delete resources to avoid incurring costs:

  1. On the CodeCommit console, choose Repositories.
  2. Choose your repository and choose Delete Repository.
  3. On the IAM console, choose Users.
  4. Choose your pipeline user and choose Delete User.
  5. On the navigation pane, choose Policies.
  6. Choose your CodeCommit Git push policy and choose Policy Actions and Delete.
  7. Go to the Pipeline page within Azure DevOps and choose your pipeline.
  8. Choose More Actions and choose Delete.

Conclusion

This post showed how you can use an Azure DevOps pipeline to mirror an Azure DevOps repository in CodeCommit. It provided detailed instructions on setting up your replica repository in CodeCommit, creating a least privilege access policy and user credentials for the pipeline in IAM, and creating the pipeline in Azure DevOps. You can use this solution to automatically replicate your Azure DevOps repositories in AWS for backup purposes or as a source to build CI/CD pipelines within AWS.

About the author

Michael Massey
Michael Massey is a Cloud Application Architect at Amazon Web Services. He helps AWS customers achieve their goals by building highly-available and highly-scalable solutions on the AWS Cloud.

0raEs0TXqsQ

Categories: DevOps