Organizations running workloads in Amazon Web Services (AWS) often must search and analyze logs to troubleshoot or investigate operations, governance, or security events. Amazon Athena enables AWS customers to search and analyze log data directly from in Amazon Simple Storage Service (Amazon S3) using standard SQL queries. Additionally, we understand that customers need a common environment for security investigation use cases that is easy to set up, deploy, and maintain.

In this article, we provide an overview of AWS Security Analytics Bootstrap, which lets customers perform security investigations on AWS service logs through an Amazon Athena analysis environment.

Overview

AWS Security Analytics Bootstrap is an open source framework designed for AWS customers who need a quick method to set up Athena and perform investigations on AWS service logs archived in Amazon S3 buckets. AWS Security Analytics Bootstrap was designed to provide AWS customers with common requirements for security use cases, such as:

The AWS Security Analytics Bootstrap tool provides AWS CloudFormation templates for creating a fully configured Athena analysis environment, including an Amazon Athena workgroup, AWS Glue databases, AWS Glue tables, and demo Athena queries. Each table schema supports common security investigation requirements, including partitioning and searches across multiple accounts, regions, and dates.

AWS Security Analytics Bootstrap uses partition projection with Amazon Athena to provide dynamic partitioning across accounts, regions, and dates without any additional infrastructure, code, or frequent maintenance. Partitioning AWS service log data by account, region, and date allows AWS customers to create targeted queries and reduce their cost and query times.

Common use cases for AWS Security Analytics Bootstrap include situations when AWS customers:

  • Want to search AWS service logs natively in AWS.
  • Currently have no security information and event management (SIEM) available or accessible.
  • Need to search logs beyond the SIEM retention period.
  • Don’t have the required AWS service logs indexed in the SIEM.
  • Need to investigate an AWS account that hasn’t been centralizing its logs.
  • Need quick and lightweight log investigation environment when starting their security journey.

AWS Security Analytics Bootstrap currently supports the AWS service logs shown in the following table:

AWS service logDescription
1AWS CloudTrailAWS API events (management and data events)
2Amazon Virtual Private Cloud (VPC) Flow LogsNetwork events
3Amazon Route 53 DNS resolver query logsDNS resolution events

The following architecture diagrams show the two ways in which AWS Security Analytics Bootstrap can be deployed. AWS Security Analytics Bootstrap supports both a single account deployment, where it is deployed in the same account as the Amazon S3 logging buckets, and a cross-account deployment, where it is deployed in a different account than the Amazon S3 logging buckets.

Illustration of workflow for: AWS Security Analytics Bootstrap supports a single account deployment, where it is deployed in the same account as the Amazon S3 logging buckets

Illustration of workflow for: AWS Security Analytics Bootstrap supports a cross-account deployment, where it is deployed in a different account than the S3 logging buckets.

Deploying AWS Security Analytics Bootstrap

Prerequisites

  • AWS service logs (for example, AWS CloudTrail, Amazon VPC Flow Logs, Amazon Route 53 resolver query logs) must be delivered to Amazon S3 buckets unmodified in their native format.
  • For cross-account deployments, bucket policies must be in place, and objects must be owned by the bucket account owner to allow cross-account access.
  • For logs encrypted via AWS Key Management Service (KMS), the AWS Identity and Access Management (IAM) principals that will be used to submit Athena queries must have permissions for kms:Decrypt and kms:DescribeKey in their IAM policy. The KMS key policy will need to grant them the same access.

Getting started

The Athena Infrastructure CloudFormation template will deploy a fully functional security analytics environment including the resources listed in the following table:

ResourceNotes
1Athena workgroupConfigured to provide encrypted output to a specified Amazon S3 location.
Includes preconfigured demo queries as named queries.
2Glue databaseContains associated Glue tables.
3Glue tablesStandardized table schemas with dynamic partitions for account, region, and date for:
CloudTrail logs
VPC Flow Logs
Route 53 DNS Resolver Logs

Deployment time is approximately 10 minutes.

Comments are provided in the CloudFormation parameters section to assist with the parameters required for deployment, and a detailed walkthrough of the deployment process is provided in the AWS Security Analytics Bootstrap Deployment Guide. AWS Security Analytics Bootstrap has been designed to allow rapid deployment; however, we recommend deploying it and getting experience using it so you’re prepared in case there is an urgent need.

AWS Security Analytics Bootstrap resources

The Athena Infrastructure CloudFormation template can be deployed by itself or in combination with any of the additional resources depending on customers’ use cases and requirements.

Resource typeResourceResource providesCleanup and removal notes
1AWS CloudFormation templateAthena Infrastructure CloudFormation templateCreates the ready-to-use Athena security analytics environment including: Athena workgroup, Glue database, Glue tables, and demo named queries. Comments are provided in the CloudFormation parameters section to walk customers through deployment, or customers can review the AWS Security Analytics Bootstrap Deployment Guide for more detail.All resources created by this template will be deleted when the CloudFormation Stack is deleted. This will not affect the source log data.
2AWS CloudFormation templateIAM roles and policies for Athena Admin and Athena AnalystCreates IAM roles and policies for a Athena Admin and Athena Analyst Roles designed according to least-privilege principals.All resources created by this template will be deleted when the CloudFormation Stack is deleted.
3AWS CloudFormation templateEnable flow logsEnables VPC Flow Logs for the specified VPC, subnet, or ENI with all fields through v5 in the order expected by Athena Bootstrap.The VPC Flow Log configuration will be deleted when the CloudFormation stack is deleted. Any logs created will need to be deleted (if desired) separately from the target S3 bucket.
4CREATE TABLE SQL statementAWS CloudTrail table schemaCreates a Glue table for CloudTrail logs partitioned by account, region, and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation template; this SQL statement can be used to create a table in an existing Athena environment for ad hoc deployment use cases. “TODO” comments are included above sections must be updated with customers’ environment details.The table can be deleted with the Athena query statement DROP TABLE <table name> (for example, DROP TABLE cloudtrail).
5CREATE TABLE SQL statementAmazon VPC Flow Logs table schemaCreates a Glue table for VPC Flow Logs partitioned by account, region, and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation template; this SQL statement can be used to create a table in an existing Athena environment for ad hoc deployment use cases. “TODO” comments are included above sections must be updated with customers’ environment details.The table can be deleted with the Athena query statement DROP TABLE <table name> (for example, DROP TABLE vpcflowlogs).
6CREATE TABLE SQL StatementAmazon Route 53 Resolver Query Logs table schemaCreates a Glue table for Route 53 DNS Resolver Logs partitioned by account, VPC ID, and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation template; this SQL statement can be used to create a table in an existing Athena environment for ad hoc deployment use cases. “TODO” comments are included above sections that must be updated with customers’ environment details.The table can be deleted with the Athena query statement DROP TABLE <table name> (for example, DROP TABLE r53dns).
7Demo Athena queriesAWS CloudTrail demo queriesDemo Athena queries for CloudTrail logs. These queries area also created in the Athena workgroup as named queries by the Athena Infrastructure CloudFormation template.N/A: No resources created
8Demo Athena queriesAmazon VPC Flow Log demo queriesDemo Athena queries for VPC Flow Logs. These queries area also created in the Athena Workgroup as named queries by the Athena Infrastructure CloudFormation template.N/A: No resources created
9Demo Athena queriesAmazon Route 53 Resolver Query Log demo queriesDemo Athena queries for Route 53 DNS Resolver Logs. These queries area also created in the Athena workgroup as named queries by the Athena Infrastructure CloudFormation template.N/A: No resources created

As development continues on this tool, the following items are currently out of scope:

  • How data is provided to Amazon S3 buckets (for example, configuration of logs).
  • Optimization of the underlying data sources (for example, merging small files, converting to Parquet/ORC columnar formats).
  • Bucket policy updates for cross-account deployments.
  • AWS KMS policy updates.

By installing AWS Security Analytics Bootstrap, AWS customers may incur charges from the following services:

Conclusion

In this post, we’ve explained how AWS Security Analytics Bootstrap framework helps AWS customers investigate common operational, governance, or security issues. The project is licensed under the Apache 2.0 License, and you can find more information about its use and how to contribute on GitHub.