Month: May 2017

AWS Powershell Tools Snippets: S3 Multipart Upload Cleanup

On By Brian OlsonIn UncategorizedLeave a comment
My company does quite a bit with AWS S3. We use it to store static files and images, we push backups to it, we use it to deliver application artifacts, and the list goes on.

When you push a significant amount of data to and from S3, you’re bound to experience some network interruptions that could stop an upload. Most of the time S3 clients will recover on their own, but there are some cases where it might struggle.

One such case is when you are pushing a large file and using S3 Multi Part Uploads. This can leave you with pieces of files sitting in S3 that are not useful for anything, but still taking up space and costing you money. We recently worked with AWS support to get a report of how how many incomplete uploads we had sitting around, and it was in the double digit terabytes!

We started looking for a way to clean them up and found that AWS recently created a way to manage these with a bucket lifecycle policy. Some details are in a doc here and there’s an example of how to create this policy on the AWS CLI towards the bottom.

We decided to recreate this functionality in Powershell using “Write-S3LifecycleConfiguration Cmdlet” to make it a little easier to apply the policy to all of the buckets in our account at once.

It took a little reverse engineering. The Write-S3LifecycleConfiguration commandlet doesn’t have many useful examples. In the end I wound up creating the policy I wanted in the AWS console, and then using Get-S3LifecycleConfiguration to see how AWS is representing the policies in their .NET class structure.

It seems to me that there are a lot of classes between you and creating this policy, but that could mean that AWS has future plans to make these policies even more dynamic and useful.

The code I came up with at the end is below. Hope it’s helpful!


$rule = new-object -typename Amazon.S3.Model.LifecycleRule;
$incompleteUploadCleanupDays = new-object -typename Amazon.S3.Model.LifecycleRuleAbortIncompleteMultipartUpload
$incompleteUploadCleanupDays.DaysAfterInitiation = 7
$rule.AbortIncompleteMultipartUpload = $incompleteUploadCleanupDays
$rule.ID = "WholeBucketPolicy"
$rule.status = "Enabled"

$prefixPredicate = new-object -type Amazon.S3.Model.LifecyclePrefixPredicate

$lifecycleFilter = new-object -type Amazon.S3.Model.LifecycleFilter

$lifecycleFilter.LifecycleFilterPredicate = $prefixPredicate

$rule.Filter = $lifecycleFilter

foreach ($bucket in get-s3bucket) {
    write-host "Bucket name: $($bucket.bucketname)"
    $existingRules = get-s3lifecycleconfiguration -bucketname $bucket.bucketname
    $newPolicyNeeded = $true;
    foreach ($existingRule in $existingRules.rules) {
            if($existingRule.ID -eq $rule.ID) {
                write-host "Policy $($rule.ID) already exists, skipping bucket"
                $newPolicyNeeded = $false;
            }
    }
    if($newPolicyNeeded) {
        write-host "Rule not found, adding"
        $existingRules.rules += $rule

        Write-S3LifecycleConfiguration -bucketname $bucket.bucketname -configuration_rule $existingRules.rules
    }
}

WannaCry: Finding where SMB is allowed in AWS

On By Brian OlsonIn UncategorizedLeave a comment
WannaCry is the latest ransomware to sweep the internet and cause lots of excitement. As occasionally happens with well publicized security events like this I got to hear a former firewall admins favorite words: “Can you please take away a bunch of network access?” What fun!

I love blocking traffic as much as the next guy, but it’s not a great idea to just change firewall rules willy nilly. You should always spend a little time thinking about the impacting and looking at what access it’s prudent to remove. In this post I’ll list a couple of the commands I used to poke around our AWS Security groups and find where SMB was allowed.
Because time was a factor, I decided to do all of this in the AWS Powershell tools. As usual, you’ll need to set default credentials and an AWS region before running any of these commands.
The first one is pretty straightforward. The sequence of commands will spit out any security groups in your region that allow SMB traffic inbound in the default ports (137-139 and 445).
Fun fact, apparently AWS built this section of the powershell tools before you could control egress permissions in a group. The inbound rules are named “IpPermissions” while the outbound rules have the more describe “IpPermissionsEgress”. Other places (like cloudformation templates) name inbound rule sets “Ingress” to match the outbound version.
get-ec2securitygroup | foreach {foreach ($ingress in $_.ippermissions) {if(($ingress.fromport -le 445 -and $ingress.toport -ge 445) -or $ingress.fromport -le 137 -and $ingress.toport -ge 139) -or $ingress.ipprotocol -eq -1){$_.GroupName}}} | select-object  -unique

I apologize for the length of this single line, but it should show you anywhere your instances are accepting SMB, including rules that allow all traffic.

Next it can also be helpful to look for rules allowing SMB outbound. You might want to restrict this traffic too. This command is almost the same as the first 

PS C:\Users\bolson\Documents\AWS> get-ec2securitygroup | foreach {foreach ($egress in $_.ippermissionsegress) {if(($egress.fromport -le 445 -and $egress.toport -ge 445) -or ($egress.fromport -le 137 -and $egress.toport -ge 137) -or $egress.ipprotocol -eq -1){$_.GroupName}}} | select-object  -unique

Now that you’ve got a listing of where SMB is allowed in your AWS account, you may want to remove specific security groups from instances. If you’re looking to do one or two instances, that can be done pretty easily through the console. If you’re looking to pull a few security groups off of every instance, you can use the example below, updating the security group IDs.

We used this example for removing a “temp setup” group that we use in our environment to allow extra access for configuring a new instance. 

set-defaultawsregion us-east-1
set-awscredentials -profilename PHI
(Get-EC2Instance -filter @( @{name='instance.group-id';values="sg-11111","sg-22222"})).instances | foreach {
    write-host "Instance Name: $(($_.tags | where {$_.key -eq "Name"}).value) - $($_.InstanceId)";
    $finalGroups = @();
    $finalGroupNames = @();
    foreach ($group in $_.SecurityGroups) {
        write-host $group.groupid
        if($group.groupid -ne "sg-11111" -and $group.groupid -ne "sg-22222") {
            write-host "$($group.groupid -ne 'sg-333333')"
            $finalGroups += $group.groupid;
            $finalGroupNames += $group.groupname
        }
    }
    Edit-EC2InstanceAttribute -InstanceId $_.InstanceId -group $finalGroups
    write-host "Finalgroups: $($finalGroupNames)"
}

Hopefully that helps you do some analysis in your environment!

Auditing AWS IAM Users

On By Brian OlsonIn UncategorizedLeave a comment
Like any other company with sensitive data we go through audits pretty regularly. The latest one included some questions about accounts that have access to sensitive data, and the number of auth factors required to log into them.

As usual I started digging around in the AWS Powershell Tools to find a way to make this job easier than just manually looking through accounts, and I quickly found Request-IAMCredentialReport and Get-IAMCredentialReport.

These two commands are a pretty interesting combination. The first tells AWS to generate a credential report. Request-IAMCredentialReport is the step required to generate the report on AWS’s end. There’s some pretty good documentation on how that section works. The most interesting point to me is that AWS will only generate a report every 4 hours. This is important to note if you’re making changes, and re-running reports to double check they fixed an issue.
The second command, Get-IAMCredentialReport actually downloads the report that’s generated. From what I’ve seen, if you haven’t run Request-IAMCredentialReport in the last 4 hours to have a fresh report, this command will fail.
I found the output of this command to be the most useful when I included the -AsTextArray option. That returns an array of lines of the report with the columns separated by columns. I won’t include an sample output from our accounts for obvious reasons, but check the documentation for an example of what that looks like.
Now that we’ve got all of the components to download this report, it’s pretty trivial powershell work to do some parsing and logic.
The script example below creates an array of maps for each line of the report, letting you iterate over it and check for different conditions.
The one I’m testing for now is IAM accounts that have passwords enabled, but do not have an MFA device activated, but you can see it would be pretty easy to add additional criteria that would be tested against the report.
$awsProfiles = @("FirstProfileName","SecondProfileName");
set-DefaultAWSRegion us-east-1;
foreach ($awsProfile in $awsProfiles) {
    write-host "Running audit on $awsProfile";
    Set-AWSCredentials -ProfileName $awsProfile;
    
    # Attempt to get an IAM credential report, if one does not exist, sleep to let one generate
    $reportResult = Request-IAMCredentialReport -Force;

    # Sleep for 15 seconds to allow the report to generate
    start-sleep -s 15

    try {

        # Get IAM Credential report
        $credReports = Get-IAMCredentialReport -AsTextArray;
    } catch {
        write-host "No credential report exists for this account, please run script again in a few minutes to let one generate";
        exit;
    }
    # Empty list that will contain parsed, formatted credential reports
    $credReportList = @();

    # Get the headings from the report
    $headings = $credReports[0].split(",");

    # Start processing the report, starting after the headings
    for ($i = 1; $i -lt $credReports.length; $i++) {

        # Break up the line of the report by commas
        $splitLine = $credReports[$i].split(",");
        $lineMap = @{};

        # Go through the line of the report and set a map key of the header for that column
        for ($j = 0; $j -lt $headings.length; $j++) {
            $lineMap[$headings[$j]] = $splitLine[$j];
        }

        # Add the formatted line to the final list
        $credReportList += , $lineMap;
    }

    # Iterate over the report, using rules to evaluate the contents
    foreach($credReport in  $credReportList) {
        # Check for users that have an active password, but not an active MFA device
        if($credReport['password_enabled'] -eq "TRUE" -and $credReport['mfa_active'] -eq "FALSE") {
            write-host "ALERT: User: $($credReport['user']) has a password enabled, but no MFA device"
        }
    }
    write-host "";
}

This script assumes you have created AWS Powershell tools profiles that match the array on the first line.

And here is some example output of users I had to go have a chat with today to activate their MFA devices.

NOTE: You may need to run this script a couple times, if you haven’t generated an IAM Credential Report in a while.