Your Cloudwatch Log Groups could look something like this:
As you might guess, after the retention time, logs are deleted.
And that’s a good approach for keeping the costs under control, but sometimes regulation mandates that logs are stored longer than this period.
Luckly, AWS allows you to export logs to S3.
But this is a manual operation, there’s no option to export logs periodically and automatically.
So what we want to achieve here is:
Every day, export logs of Log Groups to an S3 bucket
Only export logs that were not exported before
Archive the logs with S3 Glacier for even lower costs
Exporting logs of Log Groups to an S3 bucket
Let’s get to the reason you’re here. Here’s the lambda function:
At a glance:
Env variable S3_BUCKET needs to be set. It’s the bucket to export the logs.
Creates a Cloudwatch Logs Export Task
It only exports logs from Log Groups that have a tag ExportToS3=true
It will use the log group name as the prefix folder when exporting
Saves a checkpoint in SSM so it exports from that timestamp next time
Only exports if 24 hours have passed from the last checkpoint
One caveat I found was that AWS only allows one Export Task running per account. This means when the lambda function was trying to export multiple log groups at once, I got a LimitExceededException error.
To mitigate that, I added a time.sleep(5) that seems to be enough to allow an Export Task to finish and next one can start.
But, there might be cases where the Export Task takes more than 5 seconds or you have too many logs groups and the lambda execution time is not enough to export all of them.
In this case I recommend running the lambda every 4 hours. It will only export log groups that haven’t been exported for 24 hours, so it’s safe to do without causing overlapping logs to be exported.
But wait a minute. CloudWatch Log Groups doesn’t have tags!
That’s right. The AWS Console show no option to list or edit tags.
But they are there, I promise!
You can set a tag to a log group using the command:
Replace <NAME OF THIS BUCKET> with the bucket name created and <EXTERNAL ACCOUNT> with any external AWS accounts you with to allow to export logs from.
And for archiving it to Glacier, the Lifecycle rule is:
We hope this article helped you setting up a more compliant, automated and cost-efficient infrastructure.
If you are interested in infrastructure-as-code, please check out our plethora of Terraform modules, Docker images and CLIs that help our clients to automate their infrastructure at https://modules.dnx.one
We work at DNX Solutions and help to bring a better cloud and application experience for digital native startups in Australia.
Our current focus areas are: AWS, Well-Architected Solutions, Containers, ECS, Kubernetes, Continuous Integration/Continuous Delivery and Service Mesh.
We are constantly hiring cloud engineers for our Sydney office, focusing on cloud-native concepts.