VPC Flowlogs through Cloudwatch Logs Insights

You know all those times you think to yourself, “I wish there were a faster way to search all these logs I keep putting in Cloudwatch?”

Well apparently Alexa was reading your mind at one of those times because at AWS re:Invent 2018 released CloudWatch Logs Insights. It’s advertised as a more interactive, helpful log analytics solution than the log search option we already have.
This last week I got some questions about blocked traffic in our AWS account, which seemed like the perfect opportunity to give it a shot (NOTE: You will need to be sending your VPC flowlogs to Cloudwatch for this to work for you).
Here are some of the queries I tried out, most of them based loosely off of the examples page.

Count of Blocked Traffic by Source IP and Destination Port

fields @timestamp, dstPort, srcAddr
| stats count(*) as countSrcAddr by srcAddr, dstPort
| sort countSrcAddr desc

This query gives use the the top blocked senders by destination service.

Using this I pretty quickly found an ELB with the wrong instance port.

This worked for me because we separate our accept and reject flowlogs. If you keep them together you can add a filter as the first line

filter action = 'REJECT'
| fields @timestamp, dstPort, srcAddr
| stats count(*) as countSrcAddr by srcAddr, dstPort
| sort countSrcAddr desc

Blocked Destination Addresses and Ports for a Specific Source

During our troubleshooting we noticed a specific address sending us a lot of traffic on port 8088, which is vendor specific or an alternate HTTP port. This was a little odd because we don’t use 8088 externally or internally.
We dug in using this query
filter srcAddr = 'x.x.x.x'
| fields @timestamp, dstPort, srcAddr, dstAddr

Where x.x.x.x is the source IP address.

I’m not going to post the screen shot because there are a lot of our destination addresses and it would take time to block them out, but you get the idea.

We did a lookup on the address and it was owned by Digital Ocean, which is a cloud hosting company. It’s likely someone was doing a scan from a server in their environment, hard to say if it was good or bad intentions.

To satisfy my curiosity I wanted to ask the question, “When did the scan begin and how aggressive is it?”
So I added a “stats” filter to group the sum of the packets by 5 minute totals.

filter srcAddr = 'x.x.x.x' and dstPort = '8088'
| fields @timestamp, dstPort, srcAddr, dstAddr
| stats sum(packets) by bin( 30 min)

When you use the stats command with a time series you can get a “visualization” like the one below:

It looks like the scan lasted about 6 hours, from 4 am or so my time to around 9:45 my time.

Conclusion

CloudWatch Log Insights is a much faster way to analyze your logs than the current Cloudwatch search. The query language is pretty flexible, and reasonably intuitive (though I did spend several minutes scratching my head over the syntax before I found a helpful example).
While it’s an improvement over what was there, it’s not on par with a log analytics tool like Splunk, or a data visualization tool like Kibana. The visualizations page for Insights only works with time series data (as far as I can tell) and isn’t very intuitive for combining multiple query results. For that you still have to import the query into a cloudwatch dashboard.
Amazon’s edge over those more mature tools is that it’s integrated into your AWS account already, with almost no setup required (except getting your logs into Cloudwatch), and the pricing model. As usual with AWS it’s extremely friendly for getting up and started, but it’s easy to see how the cost could grow out of control if you’re not paying attention (picture someone creating a query for a months worth of data on a dashboard that refreshes every 30 seconds).
Happy querying!

AWS Powershell Tools Snippets: S3 Multipart Upload Cleanup

My company does quite a bit with AWS S3. We use it to store static files and images, we push backups to it, we use it to deliver application artifacts, and the list goes on.

When you push a significant amount of data to and from S3, you’re bound to experience some network interruptions that could stop an upload. Most of the time S3 clients will recover on their own, but there are some cases where it might struggle.

One such case is when you are pushing a large file and using S3 Multi Part Uploads. This can leave you with pieces of files sitting in S3 that are not useful for anything, but still taking up space and costing you money. We recently worked with AWS support to get a report of how how many incomplete uploads we had sitting around, and it was in the double digit terabytes!

We started looking for a way to clean them up and found that AWS recently created a way to manage these with a bucket lifecycle policy. Some details are in a doc here and there’s an example of how to create this policy on the AWS CLI towards the bottom.

We decided to recreate this functionality in Powershell using “Write-S3LifecycleConfiguration Cmdlet” to make it a little easier to apply the policy to all of the buckets in our account at once.

It took a little reverse engineering. The Write-S3LifecycleConfiguration commandlet doesn’t have many useful examples. In the end I wound up creating the policy I wanted in the AWS console, and then using Get-S3LifecycleConfiguration to see how AWS is representing the policies in their .NET class structure.

It seems to me that there are a lot of classes between you and creating this policy, but that could mean that AWS has future plans to make these policies even more dynamic and useful.

The code I came up with at the end is below. Hope it’s helpful!


$rule = new-object -typename Amazon.S3.Model.LifecycleRule;
$incompleteUploadCleanupDays = new-object -typename Amazon.S3.Model.LifecycleRuleAbortIncompleteMultipartUpload
$incompleteUploadCleanupDays.DaysAfterInitiation = 7
$rule.AbortIncompleteMultipartUpload = $incompleteUploadCleanupDays
$rule.ID = "WholeBucketPolicy"
$rule.status = "Enabled"

$prefixPredicate = new-object -type Amazon.S3.Model.LifecyclePrefixPredicate

$lifecycleFilter = new-object -type Amazon.S3.Model.LifecycleFilter

$lifecycleFilter.LifecycleFilterPredicate = $prefixPredicate

$rule.Filter = $lifecycleFilter

foreach ($bucket in get-s3bucket) {
write-host "Bucket name: $($bucket.bucketname)"
$existingRules = get-s3lifecycleconfiguration -bucketname $bucket.bucketname
$newPolicyNeeded = $true;
foreach ($existingRule in $existingRules.rules) {
if($existingRule.ID -eq $rule.ID) {
write-host "Policy $($rule.ID) already exists, skipping bucket"
$newPolicyNeeded = $false;
}
}
if($newPolicyNeeded) {
write-host "Rule not found, adding"
$existingRules.rules += $rule

Write-S3LifecycleConfiguration -bucketname $bucket.bucketname -configuration_rule $existingRules.rules
}
}

WannaCry: Finding where SMB is allowed in AWS

WannaCry is the latest ransomware to sweep the internet and cause lots of excitement. As occasionally happens with well publicized security events like this I got to hear a former firewall admins favorite words: “Can you please take away a bunch of network access?” What fun!

I love blocking traffic as much as the next guy, but it’s not a great idea to just change firewall rules willy nilly. You should always spend a little time thinking about the impacting and looking at what access it’s prudent to remove. In this post I’ll list a couple of the commands I used to poke around our AWS Security groups and find where SMB was allowed.
Because time was a factor, I decided to do all of this in the AWS Powershell tools. As usual, you’ll need to set default credentials and an AWS region before running any of these commands.
The first one is pretty straightforward. The sequence of commands will spit out any security groups in your region that allow SMB traffic inbound in the default ports (137-139 and 445).
Fun fact, apparently AWS built this section of the powershell tools before you could control egress permissions in a group. The inbound rules are named “IpPermissions” while the outbound rules have the more describe “IpPermissionsEgress”. Other places (like cloudformation templates) name inbound rule sets “Ingress” to match the outbound version.
get-ec2securitygroup | foreach {foreach ($ingress in $_.ippermissions) {if(($ingress.fromport -le 445 -and $ingress.toport -ge 445) -or $ingress.fromport -le 137 -and $ingress.toport -ge 139) -or $ingress.ipprotocol -eq -1){$_.GroupName}}} | select-object  -unique

I apologize for the length of this single line, but it should show you anywhere your instances are accepting SMB, including rules that allow all traffic.

Next it can also be helpful to look for rules allowing SMB outbound. You might want to restrict this traffic too. This command is almost the same as the first

PS C:\Users\bolson\Documents\AWS> get-ec2securitygroup | foreach {foreach ($egress in $_.ippermissionsegress) {if(($egress.fromport -le 445 -and $egress.toport -ge 445) -or ($egress.fromport -le 137 -and $egress.toport -ge 137) -or $egress.ipprotocol -eq -1){$_.GroupName}}} | select-object  -unique

Now that you’ve got a listing of where SMB is allowed in your AWS account, you may want to remove specific security groups from instances. If you’re looking to do one or two instances, that can be done pretty easily through the console. If you’re looking to pull a few security groups off of every instance, you can use the example below, updating the security group IDs.

We used this example for removing a “temp setup” group that we use in our environment to allow extra access for configuring a new instance.

set-defaultawsregion us-east-1
set-awscredentials -profilename PHI
(Get-EC2Instance -filter @( @{name='instance.group-id';values="sg-11111","sg-22222"})).instances | foreach {
write-host "Instance Name: $(($_.tags | where {$_.key -eq "Name"}).value) - $($_.InstanceId)";
$finalGroups = @();
$finalGroupNames = @();
foreach ($group in $_.SecurityGroups) {
write-host $group.groupid
if($group.groupid -ne "sg-11111" -and $group.groupid -ne "sg-22222") {
write-host "$($group.groupid -ne 'sg-333333')"
$finalGroups += $group.groupid;
$finalGroupNames += $group.groupname
}
}
Edit-EC2InstanceAttribute -InstanceId $_.InstanceId -group $finalGroups
write-host "Finalgroups: $($finalGroupNames)"
}

Hopefully that helps you do some analysis in your environment!

Auditing AWS IAM Users

Like any other company with sensitive data we go through audits pretty regularly. The latest one included some questions about accounts that have access to sensitive data, and the number of auth factors required to log into them.

As usual I started digging around in the AWS Powershell Tools to find a way to make this job easier than just manually looking through accounts, and I quickly found Request-IAMCredentialReport and Get-IAMCredentialReport.

These two commands are a pretty interesting combination. The first tells AWS to generate a credential report. Request-IAMCredentialReport is the step required to generate the report on AWS’s end. There’s some pretty good documentation on how that section works. The most interesting point to me is that AWS will only generate a report every 4 hours. This is important to note if you’re making changes, and re-running reports to double check they fixed an issue.
The second command, Get-IAMCredentialReport actually downloads the report that’s generated. From what I’ve seen, if you haven’t run Request-IAMCredentialReport in the last 4 hours to have a fresh report, this command will fail.
I found the output of this command to be the most useful when I included the -AsTextArray option. That returns an array of lines of the report with the columns separated by columns. I won’t include an sample output from our accounts for obvious reasons, but check the documentation for an example of what that looks like.
Now that we’ve got all of the components to download this report, it’s pretty trivial powershell work to do some parsing and logic.
The script example below creates an array of maps for each line of the report, letting you iterate over it and check for different conditions.
The one I’m testing for now is IAM accounts that have passwords enabled, but do not have an MFA device activated, but you can see it would be pretty easy to add additional criteria that would be tested against the report.
$awsProfiles = @("FirstProfileName","SecondProfileName");
set-DefaultAWSRegion us-east-1;
foreach ($awsProfile in $awsProfiles) {
write-host "Running audit on $awsProfile";
Set-AWSCredentials -ProfileName $awsProfile;

# Attempt to get an IAM credential report, if one does not exist, sleep to let one generate
$reportResult = Request-IAMCredentialReport -Force;

# Sleep for 15 seconds to allow the report to generate
start-sleep -s 15

try {

# Get IAM Credential report
$credReports = Get-IAMCredentialReport -AsTextArray;
} catch {
write-host "No credential report exists for this account, please run script again in a few minutes to let one generate";
exit;
}
# Empty list that will contain parsed, formatted credential reports
$credReportList = @();

# Get the headings from the report
$headings = $credReports[0].split(",");

# Start processing the report, starting after the headings
for ($i = 1; $i -lt $credReports.length; $i++) {

# Break up the line of the report by commas
$splitLine = $credReports[$i].split(",");
$lineMap = @{};

# Go through the line of the report and set a map key of the header for that column
for ($j = 0; $j -lt $headings.length; $j++) {
$lineMap[$headings[$j]] = $splitLine[$j];
}

# Add the formatted line to the final list
$credReportList += , $lineMap;
}

# Iterate over the report, using rules to evaluate the contents
foreach($credReport in $credReportList) {
# Check for users that have an active password, but not an active MFA device
if($credReport['password_enabled'] -eq "TRUE" -and $credReport['mfa_active'] -eq "FALSE") {
write-host "ALERT: User: $($credReport['user']) has a password enabled, but no MFA device"
}
}
write-host "";
}

This script assumes you have created AWS Powershell tools profiles that match the array on the first line.

And here is some example output of users I had to go have a chat with today to activate their MFA devices.

NOTE: You may need to run this script a couple times, if you haven’t generated an IAM Credential Report in a while.

AWS Powershell Tools Snippets: Powershell Pipes

I’m on another AWS Powershell tools rant. Hopefully after reading this blog post you’ll share my appreciation for how useful they are.

Powershell takes the idea of piping commands together (sending that output of one command directly to the input of another) to a whole new level of useful. If you aren’t familiar with the concept, it’s a great way to make your commands dynamic and intricate. Let’s walk through an example.

Let’s say you want to list all of your Opsworks instances in an AWS account. You’d probably start by going to the AWS Powershell Tools commandlet reference and find the Get-OPSInstances commandlet. But running it alone will give you the not-so-helpful output below
Looking a little more closely at the documentation you’ll find a “StackId” parameter that will give you instances for that stack. Obviously you could go to the console, find a stack, copy the ID, and paste it into the shell, but that feels like a lot of steps.
Instead you could use two commands, connected by a pipe to link they’re results together. For example “Get-OpsStacks” piped into 
“Get-OpsInstances”. In the command below I’m using the “count” property of powershell lists to find the number of instances returned, which is more than the zero by my first command.
Pretty cool! I was able to pull back instances without going to the console to look up a Stack ID.
This is a pretty basic example, but once you’re comfortable with the concept the possibilities are endless. Let’s say we wanted to find only Opsworks Stacks that contain the number “901” (we use the “900” range for test instances at my company). You could pipe “Get-OpsStacks” into the “Where” commandlet to filter the Opsworks stacks before you get the instances.
In the example below, I’ve also used “foreach” commandlet to only print the hostname to keep the output a little smaller.
It’s pretty clear that we’ve got instances from multiple stacks here, so we can make our where filter a little more sophisticated. Let’s find instances that are only part of the NodeJS webserver stack.
You can repeat this pattern of piping commands together as many times as you need to get the results you’re looking for. Eventually these commands can get long and complicated enough it’s better to put them into a file for some formatting and white space, but for quick and dirty work they’re pretty useful.

Building a Docker container for easy SSH into Opsworks Stacks

Part of the concept behind Opsworks is the ability to create and destroy instances dynamically. If your instances are configured by Chef recipes all the way from AMI to processing production workload, this is probably something you do pretty regularly.

But this probably means that the IP addresses behind your instances change regularly. At some point you might get tired of constantly going back to the Opsworks console to get an IP address, I know I did.

It turns out it’s not too difficult to generate an ssh config file using boto3 to pull down the instances IP addresses. I chose to do this in python, and an example script is below. In my case, our instances all have private IP addresses, so that’s the property I’m using.

import os
import boto3

ssh_config_filename = '/home/meuser/.ssh/config'
if os.path.exists(ssh_config_filename):
os.remove(ssh_config_filename)

if not os.path.exists('/home/meuser/.ssh/'):
os.mkdir('/home/meuser/.ssh/')

profiles = {'NoPHI':[{'StackName':'My-Dev-Stack','IdentityFile':'my-dev-private-key.pem', 'ShortName':'dev'}
],
'PHI':[{'StackName':'My-prod-stack','IdentityFile':'my-prod-private-key.pem', 'ShortName':'prod'}]
}

for profile in profiles.keys():
session = boto3.Session(profile_name=profile)

opsworks_client = session.client('opsworks')
opsworks_stacks = opsworks_client.describe_stacks()['Stacks']
for opsworks_stack in opsworks_stacks:
for stack in profiles[profile]:
if opsworks_stack['Name'] == stack['StackName']:
instances = opsworks_client.describe_instances(StackId=opsworks_stack['StackId'])
for instance in instances['Instances']:
with open(ssh_config_filename, "a") as ssh_config_file:
ssh_config_file.write("Host " + (stack['ShortName'] + '-' + instance['Hostname']).lower() + '\n')
ssh_config_file.write(" Hostname " + instance['PrivateIp'] + '\n')
ssh_config_file.write(" User ubuntu\n")
ssh_config_file.write(" IdentityFile " + '/home/meuser/keys/' + stack['IdentityFile'] + '\n')
ssh_config_file.write("\n")
ssh_config_file.write("\n")

This script will run through the different AWS account profiles you specify, find the instances in the stacks you specify, and let you ssh into them using

ssh dev-myinstance1

If you have an instance in your Opsworks stack named “myinstance1”. If you run linux as your working machine, you’re really done at this point. But if you’re on Windows like me, there’s another step that can make this even easier: running this script in a Docker linux container to make ssh’ing around easier.

First, you’ll need to install Docker for windows. It might be helpful to go through some of their walk throughs as well if you aren’t familiar with Docker.

Once you have the Docker daemon installed and running, you’ll need to create a Docker image from a Docker file that can run the python script we have above. I’ve got an example below of using the ubuntu:latest image, installing python, moving over your AWS secret keys and private keys for sshing to the image, and running the python script.

You will need to put the files being moved over (ssh_config_updater.py, my-prod-private-key.pem, my-dev-private-key.pem, and credentials) in the same directory as the docker file.

FROM ubuntu:latest
RUN useradd -d /home/meuser -m meuser
RUN apt-get update
RUN apt-get install -y python-pip
RUN pip install --upgrade pip
RUN apt-get install -y vim
RUN apt-get install -y ssh
RUN pip install --upgrade awscli
ADD my-dev-private-key.pem /home/meuser/keys/my-dev-private-key-dev.pem
ADD my-prod-private-key.pem /home/meuser/keys/my-prod-private-key.pem
RUN chmod 600 /home/meuser/keys/*
RUN chown bolson /home/meuser/keys/*
ADD ssh_config_updater.py /home/meuser/ssh_config_updater.py
ADD credentials /home/meuser/.aws/credentials
RUN pip install boto3
USER meuser
WORKDIR /home/meuser
RUN python /home/meuser/ssh_config_updater.py
CMD /bin/bash

Once you have your Dockerfile and build directory setup, you can run the command below with the docker daemon running.

docker build -t opsworks-manage .

Once that command finishes, you can ssh into your instances with


docker run -it --name opsworks-manage opsworks-manage ssh dev-myinstance1

This creates a running container with the name opsworks-manage. You can re-use this container to ssh into instances using

docker exec -it opsworks-manage ssh dev-myinstance1

A couple notes, I’m using the default “ubuntu” account AWS builds into Ubuntu instances for simplicity. This is a root account, and in practice you should create another account to use for normal management, either through an opsworks recipe or by using Opsworks to create the user account.

Another note, because this example copies over ssh keys and credentials files to the Docker container, you should never push this image to a container registry. If you plan on version controlling the Dockerfile, you should make sure to use a .gitignore file to keep that sensitive information out of source control.

AWS Codepipeline: Alert on Stage Failure

We’ve been using AWS Codepipeline for some time now and for the most part it’s a great managed service. Easy to get started with and pretty simple to use.

That being said, it does lack some features out of the box that most CICD systems have ready for you. The one I’ll be tackling today is alerting on a stage failure.

Out of the box Codepipeline won’t alert you when there’s a failure at a stage. Unless you go in and literally look at it in the console, you won’t know that anything is broken. For example when I started working on this blog entry, I checked one of the pipelines that delivers to our test environment, and found it in a failed state.

In this case the failure is because our Opsworks stacks are set to turn off test instances when not during business hours, but for almost any other failure I would want to alert the team responsible for making the change that failed.

For a solution, we’ll use these resources

  • AWS Lambda
  • Boto3
  • AWS SNS Topics
  • Cloudformation
First we’ll need a Lambda function that can get a list of pipelines that are in our account, scan their stages, detect failures, and produce alerts. Below is a basic example of what we’re using. I’m far from a python expert, so I understand that there are improvements that could be made for error handling.
import boto3
import logging
import os

def lambda_handler(event, context):
# Get a cloudwatch logger
logger = logging.getLogger('mvp-alert-on-cp-failure')
logger.setLevel(logging.DEBUG)

sns_topic_arn = os.environ['TOPIC_ARN']

# Obtain boto3 resources
logger.info('Getting boto 3 resources')
code_pipeline_client = boto3.client('codepipeline')
sns_client = boto3.client('sns')

logger.debug('Getting pipelines')
for pipeline in code_pipeline_client.list_pipelines()['pipelines']:
logger.debug('Checking pipeline ' + pipeline['name'] + ' for failures')
for stage in code_pipeline_client.get_pipeline_state(name=pipeline['name'])['stageStates']:
logger.debug('Checking stage ' + stage['stageName'] + ' for failures')
if 'latestExecution' in stage and stage['latestExecution']['status'] == 'Failed':
logger.debug('Stage failed! Sending SNS notification to ' + sns_topic_arn)
failed_actions = ''
for action in stage['actionStates']:
logger.debug(action)
logger.debug('Checking action ' + action['actionName'] + ' for failures')
if 'latestExecution' in action and action['latestExecution']['status'] == 'Failed':
logger.debug('Action failed!')
failed_actions += action['actionName']
logger.debug('Publishing failure alert: ' + pipeline['name'] + '|' + stage['stageName'] + '|' + action['actionName'])
logger.debug('Publishing failure alert: ' + pipeline['name'] + '|' + stage['stageName'] + '|' + failed_actions)
alert_subject = 'Codepipeline failure in ' + pipeline['name'] + ' at stage ' + stage['stageName']
alert_message = 'Codepipeline failure in ' + pipeline['name'] + ' at stage ' + stage['stageName'] + '. Failed actions: ' + failed_actions
logger.debug('Sending SNS notification')
sns_client.publish(TopicArn=sns_topic_arn,Subject=alert_subject,Message=alert_message)

return "And we're done!"

If you’re looking closely, you’re probably wondering what the environment variable named “TOPIC_ARN” is, which leads us to the next piece: A cloudformation template to create this lambda function.

The Cloudformation template needs to do a few things.

  1. Create the Lambda function. I’ve chosen to do this using AWS Serverless Application Model.
  2. Create an IAM Role for the Lambda function to execute under
  3. Create IAM policies that will give the IAM role read access to your pipelines, and publish access to your SNS topic
  4. Create an SNS topic with a list of individuals you want to get the email
The only really new fangled Cloudformation feature I’m using here is AWS SAM, the rest of these have existed for quite a while. In my opinion one of the main ideas behind AWS SAM is to package your entire Serverless Function in a single Cloudformation template, so the example below does all four of these steps.
#############################################
### Lambda function to alert on pipeline failures
#############################################

LambdaAlertCPTestFail:
Type: AWS::Serverless::Function
Properties:
Handler: mvp-alert-on-cp-failure.lambda_handler
Role: !GetAtt IAMRoleAlertOnCPTestFailure.Arn
Runtime: python2.7
Timeout: 300
Events:
CheckEvery30Minutes:
Type: Schedule
Properties:
Schedule: cron(0/30 12-23 ? * MON-FRI *)
Environment:
Variables:
STAGE_NAME: Test
TOPIC_ARN: !Ref CodePipelineTestStageFailureTopic
CodePipelineTestStageFailureTopic:
Type: "AWS::SNS::Topic"
Properties:
DisplayName: MvpPipelineFailure
Subscription:
-
Endpoint: 'pipelineCurator@example.com'
Protocol: 'email'
TopicName: MvpPipelineFailure
IAMPolicyPublishToTestFailureTopic:
Type: "AWS::IAM::Policy"
DependsOn: MoveToPHIIAMRole
Properties:
PolicyName: !Sub "Role=AlertOnCPTestFailure,Env=${AccountParameter},Service=SNS,Rights=Publish"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "sns:Publish"
Resource:
- !Ref CodePipelineTestStageFailureTopic
Roles:
- !Ref IAMRoleAlertOnCPTestFailure
IAMPolicyGetPipelineStatus:
Type: "AWS::IAM::Policy"
DependsOn: MoveToPHIIAMRole
Properties:
PolicyName: !Sub "Role=AlertOnCPTestFailure,Env=${AccountParameter},Service=CodePipeline,Rights=R"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "codepipeline:GetPipeline"
- "codepipeline:GetPipelineState"
- "codepipeline:ListPipelines"
Resource:
- "*"
Roles:
- !Ref IAMRoleAlertOnCPTestFailure
IAMRoleAlertOnCPTestFailure:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Sub "Role=AlertOnCPTestFailure,Env=${AccountParameter},Service=Lambda"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"

#############################################
### End of pipeline failure alerting lambda function
#############################################

And that’s about it. A couple notes on the Cloudformation template:

Alert Frequency

I’m using a cron expression as my schedule, currently set to go off every half hour during business hours, because we don’t have over night staff that would be able to look at Pipeline failures. You can easily up the frequency with something like

cron(0/5 12-23 ? * MON-FRI *)

Lambda Environment Variables

One of the announcements from reInvent I was most excited about was AWS Lambda environment variables. This is a pretty magical feature that lets you pass in values to your Lambda functions. In this case, I’m using it to pass an SNS topic ARN that’s being created in the same Cloudformation template into the Lambda function.

Long story short, that means we can create resources in AWS and pass references to them into code without having to have a way to search for them or put their values into source code.

      Environment:
Variables:
STAGE_NAME: Test
TOPIC_ARN: !Ref CodePipelineTestStageFailureTopic

Flowerboxes

The CFT this example comes from contains multiple pipeline management functions, so the flower boxes (“###############”) at the beginning and end of the Lambda Function definition are our way of keeping resources for each lambda function separated.

SNS Notifications

When you create an SNS topic with an email, the user will have to register with the topic. They’ll get an email and have to click the link to allow the notifications.

Snippets

These are snippets I pulled out of our pipeline management Cloudformation stack. Obviously you’ll have to put them into a Cloudformation template that references the SAM Cloudformation Transform and has a valid header like the one below:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:......

Happy alerting!

Building CodePipelines with Cloudformation: What’s my configuration?

My company started using AWS Codepipeline as a somewhat reluctant PoC. It’s not a full featured CICD service, but it is incredibly cost effective and easy to get started with. Amazon’s recent release of invoking Lambda functions makes it much more flexible.

We’ve been using Codepipeline for several months now, and with it starting to look like a longer term solution for us some of the AWS Console limitations are becoming prohibitive. For example you can’t move an action around in a stage in the console. Your only option it to delete and recreate the action where you wanted it to be.

Fortunately, most of these struggles are solved by creating your Pipelines in Cloudformation!

The AWS Cloudformation Template reference will get you started with the basics, but it leaves a lot of question marks around the different Action provider types and how to format stages and actions.

For example, we have an Action that deploys a Cloudformation stack. I had the action definition below:

            -
Name: Deploy-Stack
InputArtifacts:
-
Name: our-cft
ActionTypeId:
Category: Deploy
Owner: AWS
Version: 1
Provider: Cloudformation
Configuration:
ActionMode: CREATE_UPDATE
StackName: Our-Staging-Stack
TemplatePath: our-cft::our-template-name.yml
TemplateConfiguration: our-cft::stackConfig.json
Capabilities: CAPABILITY_NAMED_IAM
RoleArn: arn:aws:iam::/RoleName
RunOrder: 2

And I was getting a very vague error about the Cloudformation provider not being available. I opened a case with AWS Support only to find out that it was a casing issue, and the Provider needed to be “CloudFormation”.

Which was helpful, but still left me with a lot of questions like, “What casing to I need to use for Codebuild? Codedeploy? What are the Configuration items for Invoking Lambda functions?”

After spending some time searching the internet, and finding painfully few examples, I turned to my trusty friend the AWS Powershell Tools. My guess is that we can figure out what values to put into our Cloudformation template by digging out what an existing pipeline has for configuration.

It turns out the Powershell tools have a number of commandlets for interacting with Codepipeline. The one we’ll use today is Get-CPPipeline.

To find out more I ran

get-help get-cppipeline

Which shows the information below for the syntax of the command

So let’s give it a shot!

I can already guess the stages property is what I’ll be interested in next, so let’s pull one out to see what it looks like.

I know that this stage has a Cloudformation stack that it deploys, so let’s see if I can figure out what the configuration looks like from there. After some digging, I found the ActionTypeId attribute

And there it is! CloudFormation with all the correct capitalization. My guess is confirmed and this seems like a valid plan. I threw together this powershell function to print out information about an existing pipeline. Obviously you’ll need to use “Set-AWSCredentials” to pick a profile or IAM account that has access.

function print-pipeline($pipeline) {
write-host "Pipeline name: $($pipeline.Name)"
foreach ($stage in $pipeline.stages) {
write-host "Stage: $($stage.Name)"
foreach($action in $stage.Actions) {
Write-host "Action: $($Action.Name)"
write-host "Action Type ID: $($action.ActionTypeId.Category) - $($action.ActionTypeId.Provider)"
write-host "Configurations: "
write-host $action.Configuration
write-host ""
}
Write-host ""
}

}

set-awscredentials -profilename
set-defaultawsregion us-east-1

$pipeline = get-cppipeline -name

print-pipeline $pipeline

A couple notes, the Configuration attribute looks like it’s printing out as a list, but I wasn’t able to iterate over it. My guess is this is still early enough on in AWS supporting interactions with Codepipeline that some of the functions needed aren’t supported.

Lastly, here a some snippets of Action Types we use, pulled out of a couple of our templates.

Using S3 as an artifact source:

        - 
Name: Source
Actions:
-
Name: your-action-name
ActionTypeId:
Category: Source
Owner: AWS
Version: 1
Provider: S3
OutputArtifacts:
-
Name: your-output-name
Configuration:
S3Bucket: your-bucket-name
S3ObjectKey: our/s3/key.zip
RunOrder: 1

Using AWS Codebuild

          Name: Build
Actions:
-
Name: YourActionName
InputArtifacts:
-
Name: your-source-code
OutputArtifacts:
-
Name: your-compiled-code
ActionTypeId:
Category: Build
Owner: AWS
Version: 1
Provider: CodeBuild
Configuration:
ProjectName: your-code-build-project-name

Invoking a Lambda function

        -
Name: Test
Actions:
-
Name: your-action-name
ActionTypeId:
Category: Invoke
Owner: AWS
Version: 1
Provider: Lambda
Configuration:
FunctionName: your-function-name
UserParameters: paraemters-to-pass-in
RunOrder: 1

Running Opsworks Cookbooks

            -
Name: RunCookbooks
InputArtifacts:
-
Name: your-input-artifact-name
ActionTypeId:
Category: Deploy
Owner: AWS
Version: 1
Provider: OpsWorks
Configuration:
DeploymentType: deploy_app
StackId: your-stack-id
AppId: your-app-id
RunOrder: 3

Deploying an app with Codedeploy

            -
Name: your-action-name
InputArtifacts:
-
Name: your-input-artifact
ActionTypeId:
Category: Deploy
Owner: AWS
Version: 1
Provider: CodeDeploy
Configuration:
ApplicationName: your-app-name
DeploymentGroupName: your-group-name
RunOrder: 4

And lastly, creating a manual approval step

            -
Name: Approve
ActionTypeId:
Category: Approval
Owner: AWS
Version: 1
Provider: Manual
RunOrder: 1

So far I’ve been very pleased with the switch. We’ve been able to re-arrange actions and stages pretty easily, and it made creating additional stages have way fewer clicks.

AWS CLI: Table Output

Earlier today I stumbled on an AWS CLI feature I hadn’t noticed before, the “output” flag.

The default value of this flag is json, which is probably what you want most of the time. It makes it pretty easy to manipulate and pull out the data you need.
But if you are just looking for a visualization of your command you have the option of specifying “text” or “table” in the command.
aws ec2 --output table describe-instances

Text will give you a flat dump of the information, while table will format it pretty with colors and blocks. I used the table command to find an instance I wanted to use in the screen shot below.

Keep in mind that table output adds a lot of extra characters, so it isn’t a great way to view large numbers of instances at the same time, but if you’re looking for readable output, this is handy.

I wouldn’t recommend it, but you can set this to be the default in your AWS credentials file at

C:\users\\.aws\credentials

It would look like

AWS Powershell Tools Snippets: CodeBuild Cloudwatch Logs

We’ve been using AWS CodeBuild to run java maven builds almost since it came out. It’s great when it works, but when Maven has a problem it can pretty pretty difficult to sift through logs in the Cloudwatch console.

Below is an AWS Powershell Tools snippet that will pull down a cloudwatch log stream and dump it both to your console and to a file. There are a few parameters you’ll need to set

  1. LogStreamName – this should be the specific log stream you want to download. Usually this correlates to a Codebuild run
  2. LogGroupName – this will be the /aws/codebuild/
  3. OutFile location – this is where you want the files dumped
Get-CWLLogEvents -logstreamname logstreamname -loggroupname /aws/codebuild/yourprojectname).events | foreach {$message = "$($_.TimeStamp) - $($_.Message)";write-host $message; $message | out-file logfilelocation.log -append}

This command can also be used to pull down other log streams as well, you just need to swap out the same variables.

Happy troubleshooting!