Enforcing Least Privilege When Logging Lambda Functions to CloudWatch
UPDATE 2019-10-07: There is a bug in CloudFormation when outputting the LogGroup ARN. See the change below in 4. Define a Policy.
UPDATE 2020-06-16: Thanks to jplock, I have fixed an error in the ARN syntax for log-group, where a / should have been a :.
I notice that the AWS documentation and even their managed policies (e.g. AWSLambdaBasicExecutionRole) all provide users with insecure examples of how to setup permissions for their Lambda functions to emit their logs to CloudWatch Logs. The permissions are not least privilege, meaning they provide more permission to the Lambda function than are necessary and can lead to unintended consequences. Let's look at the common example given to users:
{
"Version": "",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs::01234567890:logs/*"
}
]
}
Granted, these consequences are fairly low in the context of writing to CloudWatch log streams, but there is still the possibility of a Lambda function writing to another function's log stream. In the event of an audit or incident, this error could be considered log tampering and impede an investigation. What we want to do instead is only permit the Lambda function to write to its own CloudWatch log stream.
The only way I've found to reliably get this least privilege permission to work with through the use of CloudFormation, so I will be using that below. The other side benefit of using this tactic is that you get to control the expiration date of your Lambda function's logs, whereas the default is to Never Expire and that can impact things like GDPR compliance, which mandate a data deletion policy.
The following CloudFormation snippets will be presented in the order that they will be executed by CloudFormation, to prevent any circular logic and constrain reading to just the important pieces.
1. Define an IAM role
LambdaRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "lambda.amazonaws.com"
Action: "sts:AssumeRole"
First we define the IAM role that the Lambda function will use when it is being invoked. There is nothing new here other than the fact that there are no inline policies attached to the role. This is because if the CloudWatch Logs policy is attached here, it will create a circular dependency error in CloudFormation.
2. Define a Lambda function
LambdaFn:
Type: "AWS::Lambda::Function"
Properties:
Role: !GetAtt Role.Arn
... omitted for brevity ...
Next the Lambda function is defined. Most of the details have been omitted for brevity and because they are outside the scope of this article. The only piece here is that the IAM role defined earlier is entered here so that the Lambda knows to use it when it is invoked. The shorthand function !GetAtt is used to retrieve the IAM Role's ARN.
3. Define a Log Group
LogGroup:
Type: "AWS::Logs::LogGroup"
Properties:
LogGroupName: !Sub "/aws/lambda/${LambdaFn}"
RetentionInDays: 90 # Custom log expiration? YAY!
Next the log group is created. This is part of the trick to obtaining least privilege. By default, if no log group is created manually then Lambda (given the right permissions) will automatically create a log group named after the function in the path /aws/lambda/. Since this path and Lambda function name is already known to us at this point, we can create it ourselves.
A side benefit of manually creating the log group is two-fold:
- We get to define a custom expiration time for any logs created by the Lambda function. By default there is no expiration on Lambda function logs, so this helps with compliance, cost control, and ease of monitoring.
- The Lambda function no longer needs to create the log group so the permission
lambda:CreateLogGroupcan be removed from the IAM role.
4. Define a Policy
Policy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "allow-lambda-logging"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- !GetAtt LogGroup.Arn
- !Sub
- "${Arn}:log-stream/*"
- Arn: !GetAtt LogGroup.Arn
Roles:
- !Ref LambdaRole
Lastly, the IAM policy is provided which is where the least-privilege permissions come into play. Notice that the Log group is referenced in a shorthand string interpolation function !Sub, which allows us to control where the log streams will be created. To repeat, we already know the well-known path that Lambda functions use to write log events, so we can scope the policy's permissions to just what is needed and no more. While wildcards are generally a security code smell, in this case it is required so that Lambda can create named log streams, and there is no security issue because these streams are grouped within the log group we created for it.
UPDATE 2019-10-07
The policy has been modified slightly due to a bug in CloudFormation when retrieving the return value of a CloudWatch Logs LogGroup. According to the documentation, the return value is arn:aws:logs:us-west-1:123456789012:log-group:/mystack-testgroup-12ABC1AB12A1:*, but the official documentation on ARN formats says that it is arn:aws:logs:us-west-1:123456789012:log-group:/mystack-testgroup-12ABC1AB12A1. Basically, CloudFormation adds a :* to the end of the log group ARN, causing the IAM policy to be invalid.
In the updated policy, the ARN for the LogGroup now needs to be manually constructed:
Policy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "allow-lambda-logging"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- !Sub "arn:aws:logs:${AWS::Region}::${AWS::AccountId}:log-group:${LogGroup}"
- !Sub "arn:aws:logs:${AWS::Region}::${AWS::AccountId}:log-group:${LogGroup}:log-stream:*"
Roles:
- !Ref LambdaRole
Minor nitpick: I have noticed that the AWS Lambda Web console will still complain that it doesn't have enough permissions to write to CloudWatch Logs, yet logs are being created. I presume this is because we have not given the Lambda function permission to create its own log group, which is exactly what we want.
Summary
Rolling it all up, we have the following CloudFormation template.
---
AWSTemplateFormatVersion: "2010-09-09"
Description: >
Provisions a Lambda function with least-privilege logging capabilities.
Resources:
LambdaFn:
Type: "AWS::Lambda::Function"
Properties:
Role: !GetAtt Role.Arn
...
LogGroup:
Type: "AWS::Logs::LogGroup"
Properties:
LogGroupName: !Sub "/aws/lambda/${LambdaFn}"
RetentionInDays: 90 # Custom expiration!
Role:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "lambda.amazonaws.com"
Action: "sts:AssumeRole"
Policy:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "allow-lambda-logging"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- !Sub "arn:aws:logs:${AWS::Region}::${AWS::AccountId}:log-group:${LogGroup}"
- !Sub "arn:aws:logs:${AWS::Region}::${AWS::AccountId}:log-group:${LogGroup}:log-stream:*"
Roles:
- !Ref LambdaRole
There you have it. A least privilege permission attach to a Lambda function, that can now only log events to its own CloudWatch log group and nothing else. It cannot even create new log groups, thereby reducing the attack surface further. And the fact that this is all written up in CloudFormation using infrastructure-as-code, makes this easy to deploy and reuse in other projects.
Enjoy and stay safe out there!