Basic CD With CodeBuild

Automated deployment from CodeCommit to S3 through CodeBuild/Lambda

Posted by Mike Apted on February 5, 2017

I've been meaning to start experimenting with CodeBuild since it's announcement and decided to put something basic but flexible together as a proof of concept.

The TL;DR was to create an environment with a CodeCommit repo and a push trigger. That trigger fires a Lambda, which invokes a CodeBuild project, depositing a set of the repo files into an S3 bucket.

It is possible to include these in a CodePipeline, rather than trigger a Lambda from CodeCommit, but there are a couple reasons I decided to go the Lambda route. First, the project is incredibly simple. There are no test cases, no real complex builds, and no approvals. Second, there is no artifact generated and CodePipeline will not let you select a CodeBuild project that does not produce an artifact. So Lambda+trigger it is.

Whenever possible I prefer to build and iterate in CloudFormation, as it allows me to solve problems once, and then not repeat small configuration and setup mistakes once fixed. It also means no copy and pasting of ARNs, resource names, etc..

At the start of our CloudFormation template we have the version, template description and parameters we expect:

---
AWSTemplateFormatVersion: "2010-09-09"
Description: "CD w CodeBuild: Automated deployment from CodeCommit to S3 through CodeBuild/Lambda"

Parameters:
  MyCodeCommitRepoName:
    Description: Name for the CodeCommit repository that will store the CloudFormation templates
    Type: String
    MinLength: 1
    MaxLength: 100
    AllowedPattern: "[a-zA-Z][a-zA-Z0-9._-]*"
  DeployableAssetsFolder:
    Description: The subfolder or path to deploy to S3 (allows you to ignore files in root directory or use a compiled output directory)
    Type: String
    MinLength: 1
    MaxLength: 100
    AllowedPattern: "[a-zA-Z.][a-zA-Z0-9._-]*"
    Default: .
  MyBucketName:
    Description: Name for the CodeCommit repository that will store the CloudFormation templates
    Type: String
    MinLength: 3
    MaxLength: 63
    AllowedPattern: "[a-z][a-z0-9.-]*"
  MyLambdaFunctionName:
    Description: Name for the Lambda function that will trigger the CloudBuild project
    Type: String
    MinLength: 3
    MaxLength: 63
    AllowedPattern: "[a-zA-Z][a-zA-Z0-9.-]*"
  MyCodeBuildProjectName:
    Description: Name for the CodeCommit repository that will store the CloudFormation templates
    Type: String
    MinLength: 2
    MaxLength: 255
    AllowedPattern: "[a-zA-Z][a-zA-Z0-9_-]*"

Normally I would try and avoid explicitly naming resources wherever possible, but due to some circular template refs and an unwillingness to increase the complexity at this point, I can live with it in this case. Where possible I have added constraints on the parameters (MinLength, MaxLength, etc.) that reflect the underlying AWS service naming restrictions.

The parameters should be somewhat self explanatory, given the descriptions, but we are asking for names for the CodeCommit repo, the S3 bucket we will deploy to, the CodeBuild project name, the Lambda function and an optional value for a subdirectory of the repo to deploy from. This is to allow us to keep certain files (git related, etc.) out of the deployment assets or to deploy a compiled "dist" folder if there is actually a build process to undergo.

Note: there is an assumption here that we are creating new resources for all these, not using an existing set, so names will need to be unique, etc..

Getting into the Resources section of the template we define our services. First up are our IAM policies and permissions. Our CodeBuild role allows CodeBuild to perform needed tasks like pulling from CodeCommit, pushing to S3 and creating and publishing CloudWatch log groups and streams. If you forget this last one you will have a bit of trouble debugging any issues!

Resources:
  CodeBuildRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                - "codebuild.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        -
          PolicyName: "DeployCodeCommitToS3"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource:
                  - !Join ["", ["arn:aws:logs:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":log-group:/aws/codebuild/", !Ref "MyLambdaFunctionName"]]
                  - !Join ["", ["arn:aws:logs:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":log-group:/aws/codebuild/", !Ref "MyLambdaFunctionName", ":*"]]
              -
                Effect: "Allow"
                Action:
                  - "codecommit:GitPull"
                Resource: !GetAtt [CodeCommitRepo, Arn]
              -
                Effect: "Allow"
                Action:
                  - "s3:PutObject"
                  - "s3:List*"
                Resource:
                  - !Join ["", ["arn:aws:s3:::", !Ref "S3Bucket"]]
                  - !Join ["", ["arn:aws:s3:::", !Ref "S3Bucket", "/*"]]

Then we have our Lambda permission and role, which allows this Lambda to be invoked by CodeCommit, and the Lambda to itself invoke a CodeBuild project's build process. Again, we make sure to include creating and publishing CloudWatch log groups and streams:

  CodeCommitLambdaPermission:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt [DeploymentLambda, Arn]
      Principal: "codecommit.amazonaws.com"

  LambdaRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                - "lambda.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      Policies:
        -
          PolicyName: "DeployCodeCommitToS3"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource:
                  - !Join ["", ["arn:aws:logs:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":log-group:/aws/lambda/", !Ref "MyCodeBuildProjectName"]]
                  - !Join ["", ["arn:aws:logs:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":log-group:/aws/lambda/", !Ref "MyCodeBuildProjectName", ":*"]]
              -
                Effect: "Allow"
                Action:
                  - "codebuild:StartBuild"
                Resource:
                  - !Join ["", ["arn:aws:codebuild:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":project/", !Ref "MyCodeBuildProjectName"]]

Next up is our S3 bucket, where the files from the CodeCommit repo will be sync'd to:

  S3Bucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName:
        Ref: MyBucketName

After that we have our Lambda function, which is a short Python snippet included inline, that takes the customData passed in the CodeCommit trigger and uses that to invoke the CodeBuild build job:

  DeploymentLambda:
    Type: "AWS::Lambda::Function"
    Properties:
      FunctionName: !Ref "MyLambdaFunctionName"
      Code:
        ZipFile: |
          import json
          import boto3

          client = boto3.client('codebuild')

          def lambda_handler(event, context):
            print("Received event:" + json.dumps(event, indent=2))
            response = client.start_build(
              projectName=event['Records'][0]['customData']
            )
            return "Build triggered"
      Description: Lambda function used by CodeCommit trigger to kick off CodeBuild project
      Handler: index.lambda_handler
      Role: !GetAtt [LambdaRole, Arn]
      Runtime: python2.7
      Timeout: 60

Following that we have our CodeCommit repo definition, which includes the trigger to call our Lambda when a push is received to the master branch. You can expand the complexity here, if desired, but this works for my current use case:

  CodeCommitRepo:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName:
        Ref: MyCodeCommitRepoName
      RepositoryDescription: CodeCommit repository that will store the CloudFormation templates
      Triggers:
        -
          Branches:
            - master
          CustomData:
            Ref: MyCodeBuildProjectName
          DestinationArn: !GetAtt [DeploymentLambda, Arn]
          Events:
            - updateReference
          Name: !Join ["-", [!Ref "MyCodeCommitRepoName", !Ref "MyCodeBuildProjectName", !Ref "MyBucketName"]]

Lastly we have the CodeBuild project which produces no artifacts, and uses an inline buildspec.yaml to sync our configured subdirectory to our S3 bucket using Boto3 (included in our build environment in the install phase):

  CodeBuildProject:
    Type: "AWS::CodeBuild::Project"
    Properties:
      Artifacts:
        Type: NO_ARTIFACTS
      Description: CodeBuild that will sync the CloudFormation templates to S3
      Environment:
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/python:2.7.12
        Type: LINUX_CONTAINER
      Name:
        Ref: MyCodeBuildProjectName
      ServiceRole:
        Ref: CodeBuildRole
      Source:
        Location: !GetAtt [CodeCommitRepo, CloneUrlHttp]
        Type: CODECOMMIT
        BuildSpec: !Sub |
          version: 0.1
          phases:
            install:
              commands:
                - pip install -U boto3
            build:
              commands:
                - aws s3 sync ${DeployableAssetsFolder} s3://${MyBucketName} --delete
            post_build:
              commands:
                - echo ***** Build completed *****

Now that our entire pipeline is defined in CloudFormation stack we can kick off it's creation either in the CloudFormation web console or on the CLI with:

aws cloudformation create-stack --stack-name DevSolCloudFormation --template-body file://codecommit2s3.yaml --parameters ParameterKey=MyCodeCommitRepoName,ParameterValue=DevSolCloudFormation ParameterKey=MyBucketName,ParameterValue=devsol-cloud-formation ParameterKey=MyCodeBuildProjectName,ParameterValue=DevSolCloudFormation ParameterKey=DeployableAssetsFolder,ParameterValue=templates ParameterKey=MyLambdaFunctionName,ParameterValue=DevSolCloudFormation --capabilities CAPABILITY_IAM

Note that we need the --capabilities CAPABILITY_IAM given the roles and permissions we are creating. Once the stack creation is complete you can then check out the CodeCommit repo locally, add files (including the subdirectory if you specified one), and any push back to the repo will kick off a CodeBuild process deploying your files to your S3 bucket.

Taking this basic process to the next level could involve things like:

  • adding a build process (say a Jekyll website build) and deploying only the subdirectory
  • adding a test component to the build process
  • adding notifications through SNS for status updates on various components
  • updating the S3 bucket to provide static website hosting or more specific policies
  • adding a CloudFront CDN to the stack fronting the bucket hosted website