Enforcing Least Privilege When Logging Lambda Functions to CloudWatch

UPDATE 2019-10-07: There is a bug in CloudFormation when outputting the LogGroup ARN. See the change below in 4. Define a Policy. UPDATE 2020-06-16: Thanks to jplock, I have fixed an error in the ARN syntax for log-group, where a / should have been a :. I notice that the AWS documentation and even their managed policies (e.g. AWSLambdaBasicExecutionRole) all provide users with insecure examples of how to setup permissions for their Lambda functions to emit their logs to CloudWatch Logs. The permissions are not least privilege, meaning they provide more permission to the Lambda function than are necessary and can lead to unintended consequences. Let’s look at the common example given to users: ...

September 20, 2019 · 6 min · 1089 words · Scott Brown

Improved Iterative CloudFormation Infrastructure Development

I love using CloudFormation for provisioning AWS services but one of the more annoying aspects about working with it is that setting up the initial CloudFormation stack is an all-or-none endeavour. This is in contrast to when a stack exists and is being updated, where changes are reversible and the stack is almost always left in a working state. This annoyance is compounded when I write a stack definition with multiple resources and, as it inevitably happens, there is a typo somewhere. CloudFormation dutifully creates all the resources, trips on the error, and rolls back the entire stack to nothingness. Then I need to delete the stack and start again instead of issuing an update or changeset request. It’s a time-sink and I wanted to find a better way. ...

September 19, 2019 · 2 min · 420 words · Scott Brown

Make Amazon Host Your Lambda Code

A common pattern I see used by Engineering teams when I provide security consulting is them creating Lambda function and hosting their code in their own S3 buckets. This S3 bucket means the Engineering team needs to secure the bucket, which means the following controls are active and maintained: no public S3 access (bucket or object) access logging (logs are sent to yet another bucket!) default encryption of all objects access control and monitoring backups failover region segregation of code written by different departments …and that’s just the start. All of that is tedious and creates security busy-work, not to mention you are still responsible for those code assets. Since AWS is hosting my Lambda function, they can host my code too. Here’s how you do it. ...

July 14, 2019 · 3 min · 482 words · Scott Brown

Implementing a Double-Lock for IAM Role Switching

IAM provides a way for users and roles to become another role. This is known as IAM role switching and uses the underlying sts:AssumeRole action. You can restrict IAM role switching in one of two ways, what I like to call the single lock and double lock methods. With any IAM role switch, there involves a two-way handshake. The person (source) switching to the role (target) must be allowed to assume the role, plus the target must allow the source to assume it. That way, an IAM role switch can be used to switch between roles within the same account, or roles within different AWS account (maybe one that you don’t even own). ...

January 24, 2019 · 3 min · 601 words · Scott Brown

Forget SSH on AWS, Use SSM SessionManager

I don’t often talk of employer, mainly to keep an arm’s length distance between them and the writing on my blog. However, one of the great things about working at Unbounce is the concept of a Professional Development day (Pro-D). This is just like when you were in school and the teachers would take a day for themselves to improve. At Unbounce, every employee gets one day (8 hours) every 2 weeks to educate themselves and elevate their professional and career interests. Some of my best ideas have come out of things I learned on Pro-D day. Today, I decided to take a moment to learn about the AWS Systems Manager Session Manager (whoa, that’s a mouthful). ...

September 21, 2018 · 9 min · 1725 words · Scott Brown

Enabling an AWS IAM MFA via CLI

NB: Linebreaks (\) have been added to the CLI commands for readability. I am in the process of setting up an AWS account for my family and part of that initial setup is to create users and roles for family members. Everyone receives readonly privileges and has to assume an IAM to gain elevated privileges. The people who are allowed to assume these elevated privileges must have a valid MFA session. ...

May 26, 2018 · 2 min · 320 words · Scott Brown

Determining Your Current EC2 Region in Go

If you program in Go and use it to access AWS resources, you’ll notice that each of the services in the SDK require a region to be explicitly specified. This is because AWS constructs an API endpoint that is region-specific and some company resources may exist in a specific region (not necessarily where the code is being run). If the code is knowingly trying to access AWS resources in the same region, it can be annoying to hardcode a region into the code. It makes the code rigid and brittle (non-portable). There is a way to have your code automatically determine its region. ...

May 26, 2018 · 2 min · 246 words · Scott Brown

Emitting UserData Events With Bosky

This article assumes intermediate knowledge of an EC2 instance’s lifecycle and various AWS services. When an Linux EC2 instance starts up, user data runs as part of the cloud-init system. This allows system administrators to configure an EC2 instance at runtime, exactly once, as user data does not automatically run ever again once the server has started. At Unbounce, we built our user data to install our services onto the machine at runtime, then configure it for the specific environment (production, staging, etc.) that it required. One issue was what to do with failures within the user data script. When a failure happens and fails to start the service successfully, the machine (if behind an auto-scaling group) will be terminated and we lose why the user data failed. The first fix for this is to use off-box logging, like Cloudwatch Logs or, in our case, SumoLogic. That helped, but the turnaround time between the log service receiving the log entry and developers being notified can reach upwards of 5 minutes. By that time, the box is dead and gone. And this all assumes that enough user data ran to configure and start the off-site logging service successfully. ...

August 9, 2017 · 4 min · 828 words · Scott Brown

A Decoupled Event Bus with CloudWatch Events

During beer o’clock at work, I happened upon Roman, who is a software developer on our API and Integrations team. He asked for my opinion on creating a sort-of “contract” between teams when setting up SNS topics and subscribing them to SQS queues. Now, this is how conversations with me often start. I’m a software developer by trade, but I have my feet in cloud infrastructure and security as well so, at the very least, I’m a good sounding board for people’s architecture ideas. I pushed a bit deeper and he finally relented and stated that he doesn’t want to think about the infrastructure, only the contract between teams and, really, he wants to emit an event and have it consumed by someone else… if they care enough to consume it. ...

July 21, 2017 · 8 min · 1642 words · Scott Brown

Transferring CodeCommit Repositories Between Regions

Recently, the CodeCommit service from AWS became available in Canada (ca-central-1 region). As I’m Canadian, I like to keep my hosting as close to home as possible, for myriad reasons, but mainly because each commit getting appended with “eh!”. The CodeCommit service from Amazon (AWS) hosts Git repositories. That’s about it. It’s no Github or Gitlab, but it does ensure at-rest encryption and access is limited to specific IAM users. If you just need private Git hosting for free (or on the cheap) without any WebUI help, CodeCommit is good. ...

June 29, 2017 · 2 min · 339 words · Scott Brown