Cloud computing introduced a drastic shift when it was first introduced in the early 2000s. Providers such as Amazon Web Services (AWS) finally made it possible to rent compute, storage and other web related services on a monthly or yearly subscription basis. Getting projects into production was a matter of a credit card swipe and a deployment. Contrasting this with the usual, months long capacity planning and procurement processes shows why cloud computing was such a game changer in the tech industry.
As a result more and more developers and companies adopted cloud computing as their go-to solution to host their applications in an easy, future proof and cost effective way.
Given that AWS started in the early 2000s it’s easy to imagine how applications grew within the AWS ecosystem and how much data such applications generated, processed and accumulated. The big data trend turned the attention to the value in the data. One of AWS reactions to this trend was the introduction of the Lambda compute service back in 2014.
The very first event sources AWS introduced for Lambda was support for events generated by S3 buckets. To this day the infamous „Image resizer“ demo where an image is uploaded to an S3 bucket and automatically resized by a Lambda function is still the most prominent „Hello World“ application to showcase the power of serverless architectures.
Existing event sources
Our dive into the AWS history and Serverless computing above showed us that AWS Lambda was initially built around the idea of working with data from existing infrastructure components such as storage buckets.
Once API Gateway support as a Lambda event source was announced it was clear that the serverless paradigm will be huge. The JAWS CLI tool (The Serverless Framework precursor) was born in 2015 out of the necessity to provide an easy way to leverage the power serverless unleashes by making it as simple as possible to author serverless applications.
In 2016, the Serverless team sat down and worked together with the community to define the integral parts for our upcoming Serverless Framework v1 release. One main takeaway during this exercise was that we should support CloudFormation which is the de-facto standard to define and deploy infrastructure on AWS in a declarative way.
While CloudFormation gives us a lot of upsides, mainly in the form of a battle-tested and well-known platform to perform resilient infrastructure changes, it not short of downsides as well. One such downside is the focus on infrastructure creation and teardown which means that CloudFormation was never designed to deal with resources which were not part of the initial creation phase.
It’s clear that this restriction prevents CloudFormation users from introducing existing infrastructure components such as S3 buckets or Cognito User Pools into their current setup. This problem trickles down to the Serverless Framework which utilizes CloudFormation under the hood.
Introducing event support for existing S3 buckets or Cognito User Pools turned into a problem only Serverless Plugins were able to solve. Such plugins relied on raw AWS SDK calls, and their usage introduced a second method to allow the management of application infrastructure.
Within recent versions of the Serverless Framework, we finally found a way to support existing resources via CloudFormation which means that we still get all the benefits CloudFormation has to offer while making it possible to introduce existing, potentially legacy infrastructure components, into the application stack.
We already read above that S3 is one of the first and most widely used AWS services. Given that S3 has been around for such a long time, it’s easy to imagine that a lot of data was accumulated in S3 buckets throughout the years. S3 buckets are often used as the central object store to save and retrieve files such as images, personal documents or log files.
S3 buckets can emit a variety of different event notifications such as
s3:ObjectCreated:* which can be used to get notifications when a new object is uploaded to the bucket or
s3:ObjectRemoved:* which will send a notification when an object was deleted from the bucket. Different
suffix configurations make it possible to further filter down when such events should be emitted.
The Serverless Framework comes with native support for the
s3 event source. Let’s take a look at an example where we want our
notify function to be called whenever a new
documents directory of our
service: acme-cloud provider: name: aws runtime: nodejs10.x functions: notify: handler: handler.hello events: - s3: bucket: acme-cloud-storage events: s3:ObjectCreated:* rules: - suffix: .pdf - prefix: documents
Deploying this service with the Serverless Framework will create the
acme-cloud-storage bucket and sets up the bucket configuration to call the
notify function accordingly.
What if our
acme-cloud-storage bucket is already in production and we want to setup our
notify Lambda function after the fact?
Thanks to our recent changes this can be achieved via the
existing: true flag:
service: acme-cloud provider: name: aws runtime: nodejs10.x functions: notify: handler: handler.hello events: - s3: bucket: acme-cloud-storage events: s3:ObjectCreated:* rules: - suffix: .pdf - prefix: documents existing: true
The Serverless Framework will automatically detect that the
acme-cloud-storage bucket was already created previously and will just setup the configuration on the bucket. Once deployed our
notify function will be called whenever files that match our rules are uploaded to the existing bucket.
Cognito User Pools
Sometimes it’s useful to react to specific user behavior within your application. A typical user-related use case is the sending of “Welcome” or “Getting started” emails whenever a user decides to sign up for your service.
Cognito User Pools offer an easy way to create customized workflows via Triggers which can be setup on User Pools. Triggers include
CustomMessage and more.
The Serverless Framework supports Cognito User Pools natively. Let’s create a service in which a
greeter Lambda function is invoked whenever a new user signup is registered in our
AcmeUsers User Pool:
service: acme-user-mgmt provider: name: aws runtime: nodejs10.x functions: greeter: handler: users.handler events: - cognitoUserPool: pool: AcmeUsers trigger: PreSignUp
Deploying this service will create the
AcmeUsers Cognito User Pool and sets up the configuration so that the
greeter function is called whenever a new user signs up for our application.
Given that the service definition above will always attempt to create a new Cognito User Pool it seems to be hard to just setup the trigger configuration without interfering the production system.
Thanks to the recent changes we made it’s dead simple to setup trigger configurations on existing Cognito User Pools. The only configuration which needs to be added is the
existing: true flag. This flag tells the Serverless Framework to skip the Cognito User Pool creation phase and instead configure the triggers on the existing one:
service: acme-user-mgmt provider: name: aws runtime: nodejs10.x functions: greeter: handler: users.handler events: - cognitoUserPool: pool: AcmeUsers trigger: PreSignUp existing: true
Using this service setup makes it possible to introduce a previously created Cognito User Pool (possibly running in production for a while) into our current serverless setup.
Using event-driven application patterns with existing infrastructure components such as S3 buckets or Cognito Users Pools is not a rare circumstance.
Given that the Serverless Framework is built on top of CloudFormation to provide production-grade reliability and resilience it was only possible to introduce external event sources via Serverless Plugins. We thought hard about potential solutions to mitigate this shortcoming. While working on the solution one of our main goals was to not give up the benefits CloudFormation provides us.
Recently we finally found a way to integrate support for existing infrastructure components into core. The first event sources for which we added this support include S3 buckets and Cognito User Pools. Setting them up requires just one line of code:
How are you using this new feature with the Serverless Framework? Share your thoughts and ideas in the comments below or tweet us @goserverless.