Automated Ledger Summaries By Email
A long time ago I wrote about my method of book-keeping using Ledger CLI. Nearly 10 years later, I'm still using it track my finances down to the penny. It's an amazing tool, and has helped me identify problematic spending habits. But it does have one achilles heel, it's not very friendly to non-technical people.
I share my finances with my family because I want to teach financial literacy and help them feel like they are a part of any decision-making activities related to money. Given that Ledger is text-based, it means they don't need to install any programs. But because it is text-based, it means they need to learn to use a text editor and the terminal (I could import it into things like Gnu Cash, but I'm not going down that rabbit hole). To improve this accessibility problem, I devised an automated notification scheme whereby once I finish updating the ledger, a summary is emailed to everyone. Here's how I did it.
Everything I wrote is using AWS, and fits well within the free tier. Do your own research to verify whether your usage of this solution fits within the free tier before starting.
The workflow is this:
- I make changes to the ledger.
- I commit the results to the git repository.
- I push the git repository to AWS CodeCommit.
- CodeCommit sends the push event to EventBridge.
- EventBridge Rule matches the CodeCommit push event and starts the CodeBuild project.
- CodeBuild runs
- CodeBuild downloads the contents of the ledger's Git repository
- CodeBuild generates a summary
- CodeBuild emails the summary to the family email address.
- CodeBuild writes the logs to a CloudWatch Logs group.
All of this infrastructure is created in CloudFormation to simplify the provisioning and also to cut down on the number of services needed (this could easily be translated into Terraform, but now you need to work with Terraform).
The CloudFormation stack looks like this.
---
AWSTemplateFormatVersion: 2010-09-09
Description: >
Provisions the ledger repository and automated email summaries.
Resources:
Repository:
Type: "AWS::CodeCommit::Repository"
Properties:
RepositoryDescription: >
Financial Ledger
RepositoryName: "financial-ledger"
SummaryBuild:
Type: "AWS::CodeBuild::Project"
Properties:
Description: >
Emails a summary of the ledger whenever the repository
gets a push to master.
Artifacts:
Type: NO_ARTIFACTS
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/standard:5.0
ImagePullCredentialsType: CODEBUILD
Type: LINUX_CONTAINER
ServiceRole: !GetAtt CodeBuildRole.Arn
Source:
Type: "CODECOMMIT"
Location: !GetAtt Repository.CloneUrlHttp
GitCloneDepth: 1
SourceVersion: "refs/heads/master"
Cache:
Type: "NO_CACHE"
TimeoutInMinutes: 10
QueuedTimeoutInMinutes: 60
LogsConfig:
CloudWatchLogs:
Status: "ENABLED"
GroupName: !Ref CodeBuildLogGroup
StreamName: !GetAtt Repository.Name
CodeBuildLogGroup:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 5
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
Description: >
Provides the CodeBuild Project with the permissions it needs.
Path: "/service-role/"
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Action: sts:AssumeRole
AllowLoggingPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: 'allow-sending-logs'
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CodeBuildLogGroup}"
- !GetAtt CodeBuildLogGroup.Arn
Roles:
- !Ref CodeBuildRole
AllowRepoClonePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: 'allow-cloning-repository'
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: codecommit:GitPull
Resource: !GetAtt Repository.Arn
Roles:
- !Ref CodeBuildRole
AllowSendingEmailPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: 'allow-send-email'
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: ses:SendEmail
Resource: !Sub "arn:aws:ses:${AWS::Region}:${AWS::AccountId}:identity/family@example.com"
Roles:
- !Ref CodeBuildRole
EventPermission:
Type: AWS::IAM::Role
Properties:
Description: >
Provides Event rules the permission to start specific code build
projects.
Path: "/service-role/"
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sts:AssumeRole
StartBuildPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: 'allow-starting-build'
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: codebuild:StartBuild
Resource: !GetAtt SummaryBuild.Arn
Roles:
- !Ref EventPermission
EventRule:
Type: AWS::Events::Rule
Properties:
Description: >
Watches for changes to the financial-ledgers repository
and starts a CodeBuild project to email a summary report.
State: ENABLED
Targets:
- Id: codebuild-project
Arn: !GetAtt SummaryBuild.Arn
RoleArn: !GetAtt EventPermission.Arn
RoleArn: !GetAtt EventPermission.Arn
EventPattern:
source:
- "aws.codecommit"
detail-type:
- "CodeCommit Repository State Change"
resources:
- !GetAtt Repository.Arn
detail:
event:
- "referenceCreated"
- "referenceUpdated"
referenceType:
- "branch"
referenceName:
- "master"
Outputs:
CloneUrlSsh:
Value: !GetAtt Repository.CloneUrlSsh
Description: >
URL used to clone the git repository over SSH.
Export:
Name: "codecommit:repository:ledger:personal"
SummaryBuildName:
Value: !Ref SummaryBuild
Description: >
Name of the CodeBuild project that sends summary emails.
Let's examine each resource in detail.
AWS::CodeCommit::Repository
This is the Git repository, and it only requires a name to get things started. Everything else can be left as a default. Why not use GitHub? Because these are financial ledgers and I want to feel secure knowing that (a) the repository is encrypted with an encryption key known only to me, (b) it is stored in a region of my choosing, and (c) it is not accessible by anyone else (AWS being the exception, but they have controls in place to prevent this access).
I also chose CodeCommit because it integrates nicely into the AWS ecosystem, and I'm exploiting this fact to get automation working seamlessly.
AWS::CodeBuild::Project
This is where the majority of the automation lives for working with Ledger. We are ~~abusing~~ extending the CodeBuild service to act as a CI workflow. In essence, we aren't "building" or compiling anything, but rather we need a serverless tool that follows a set of procedures a step-by-step until it finishes, then it cleans up after itself.
CodePipeline, which is a CI service, is not used here because CodeBuild does what it needs to do, and that thing is "just run this script". If the automation required more logic, then I would look at using CodePipeline. As it is, CodeBuild does exactly what we need for this simple automation.
CodeBuild is also useful because it hooks into the event bus underlying all AWS services. Thus, we can write some extra notification and automation logic into the CloudFormation stack to ensure that the CodeBuild project we've defined will start only when we want.
The CodeBuild project will implicitly look for a BuildSpec (the set of steps to follow) in its project configuration or, by default, in the CodeCommit repository in a file named buildspec.yml. i'll examine the contents of this file later.
The rest of the project configuration is ensuring we're using the right Docker container, and giving it the right amount of timeouts (so it doesn't incur cost overruns).
AWS::Logs::LogGroup`
A custom CloudWatch Logs group is created that will store the build logs from CodeBuild. By creating this ourselves, we can control how long the logs are stored for. Since the log data isn't sensitive, I am not encrypting it with a customer-managed KMS key, but the logs are encrypted with an AWS-managed key.
We need this log in the event that the build fails. It will explain what happened within the CodeBuild project. For 99% of the time, we don't need to look at it, so the data is retained for a very short period of time.
AWS::IAM::Role for the CodeBuild Project
An IAM role is needed for the CodeBuild project to access other AWS services within the ecosystem. From here, several IAM policies will be attached to it, providing it with the least privilege permissions to get its job done.
These policies are not added inline to the IAM role simply because it looks nicer to abstract them into separate resources.
AWS::IAM::Policy for the CodeBuild IAM Role
These are the permissions attached to the IAM role used by CodeBuild. They allow the CodeBuild project to do everything is needs to do to complete the Buildspec, plus report on its progress.
Thus, allowing logging to a CloudWatch Logs group is necessary. Plus retrieving the project's source code from CodeCommit. And lastly the ability to send emails via the AWS SES service.
I should note here that this CloudFormation stack relies on the email address used in the project to be verified within AWS SES first. This happens once, outside of the CloudFormation stack. If it is not verified, an error is produced within the CodeBuild project's logs and the project will fail.
AWS::IAM::Role for the EventPermission
Another IAM role is required in this infrastructure, this time to hold the permissions for the EventBridge Event Rule to invoke the CodeBuild project. The policies for the IAM role are specified in their own resource block.
AWS::IAM::Policy for the EventPermission
This IAM policy is attached to the EventPermission IAM role, and it grants the permission for the EventBridge Event Rule to invoke the CodeBuild project. If this isn't present, EventBridge will silently fail and the CodeBuild project won't start. I really wish EventBridge had the concept of an invocation log to hold the failures. Otherwise, you just have to know what's happening and acts as a shibboleth between those that have run event-drive infrastructure and those that haven't.
AWS::Events::Rule
This is the rule that pattern matches the event within EventBridge coming from CodeCommit, and then passes onto CodeBuild. In effect, this is the serverless "glue" that binds the event-based automation between the two services.
Outputs Section
The stack outputs some variables that will be useful for Makefile automation, especially since many of the names are automatically generated by CloudFormaiton (i.e. not guessable). The CodeCommit repository name is exported as a specific variable, for even simpler discovery. This means any script can auto-discover what outputs are in the stack by running the command aws cloudformation list-exports.
Now that the CloudFormation is ready, the CodeBuild project needs the Buildspec to tell the project what steps to run within the CodeBuild docker container. I put the buildspec.yml file inside the CodeCommit repository, simply because I want to be able to change the build specification more often than I want to change the infrastructure. This is one of those "six of one, half-dozen of the other" situations and you just have to choose one and go with it.
The buildspec.yml for CodeBuild looks like this.
---
version: 0.2
env:
variables:
EMAIL_FROM: family@example.com
EMAIL_TO: family@example.com
EMAIL_SUBJECT: Financial Summary
phases:
pre_build:
commands:
- apt update -qq
- apt install ledger -y
build:
on-failure: ABORT
commands:
- ledger -f financial.ledger equity -e "$(date +%Y/%m/%d)" > summary.txt
post_build:
commands:
- aws ses send-email --from "$EMAIL_FROM" --to "$EMAIL_TO" --subject "$EMAIL_SUBJECT" --text file://summary.txt --region $AWS_DEFAULT_REGION
Let's dive into what is happening here by focusing on the top-level attributes.
env
Some project-specific environment variables are setup in order to reduce the duplication of values in the upcoming phases. Otherwise, making changes would require hunting for all instances of the value, getting it wrong, and then iterating toward perfection. Instead, use an env variable and be happy. Your future self will thank you for putting the important configuration items at the top of the build specification file.
The environment variables being setup are where the email is going, who it is from, and what the email should be called.
phases
This is where the meat of the CodeBuild project is. Recall that CodeBuild is meant for CI processes, but it is being extended here to be a serverless runner of things. The phases are broken up into specific goals I want to attain during the project. That's because each phase is logically separated from the other: gather dependencies, process the ledger, email the result.
The pre_build section installs all the various applications we need in the standard Linux container, namely the latest version of Ledger CLI.
The build section runs the ledger command on the current year's ledger file and uses the equity command to display the account balances as entries. It ensures that no future transactions are part of the display by filtering out anything after the date when the project is run. The standard output is saved into a summary.txt file. This is the file that is being emailed.
The post_build section has one job. Send an email to the appropriate email address and include the contents of the summary.txt file in the body of the email.
When the last phase completely successfully, the project signals that it has completed successful and destroys itself and any artifacts that was kept in the Docker container. The logs are kept for several days, but then those are deleted automatically too. There is basically no maintenance needed in this infrastructure.
I hope you enjoy this automation and how it can improve visibility in your finances to everyone. You are letting computers do most of the leg work for you, all for free, securely, and with little to no artifacts laying around. More importantly, you get to focus on the important part, which is keeping your ledger up to date.