Sometimes you just need a logging bucket.
Chances are, you are working with one of the following AWS services and want to store their logs in an AWS S3 Bucket
That's what logging buckets are for.
The Nature of a Logging Bucket
A logging bucket is an AWS S3 Bucket which is accessible by the "special" AWS Log Delivery Group. For this, it needs to at least grant the
log-delivery-write permission in the ACL of the S3 Bucket.
As a best practice, it should also have server side encryption enabled and be tagged.
Creating a Logging Bucket
There are many ways to create logging buckets. The AWS console provides an option to create and configure logging buckets and CloudFormation can be used for the job as well.
In general, Terraform seems to be well suited for the job of programmatic infrastructure deployment. This is what we are going to use here.
aws_log_bucket Terraform Module
Before you can use the Terraform module out of the box, you need
After applying the Terraform module you get an S3 bucket which can be used to store AWS service logs.
The input variables for the module are defined in https://github.com/dumrauf/aws_log_bucket/settings/example.tfvars as
region = "us-east-1" shared_credentials_file = "/path/to/.aws/credentials" profile = "<your-profile>" log_bucket_prefix = "<your-prefix>-"
Here, you need to replace the example values with your settings. Note that you also need to update the
log_bucket_prefix as the current value is not a valid input.
Initialise Terraform by running
As a best practice, create a new workspace by running
terraform workspace new example
The logging bucket can then be planned by running
terraform plan -var-file=settings/example.tfvars
and created by running
terraform apply -var-file=settings/example.tfvars
The module has two outputs, namely
bucket_domain_name which are the corresponding Terraform attributes of the newly created logging bucket.
The logging bucket can be deleted by running
terraform destroy -var-file=settings/example.tfvars
Note that the actual logs in the logging bucket are not deleted by default. Hence, if the logging bucket still contains logs, deletion will fail. This is on purpose, as it requires you to remove the log files before being able to delete the logging Bucket.
AWS Logging Overview
For a detailed overview of AWS logging, see the excellent article on https://logmatic.io/blog/everything-you-need-to-know-about-aws-logging/.