Shameless self plug here, I was recently featured on the AWS blog:
https://aws.amazon.com/blogs/mt/how-docutap-automates-cloud-operations-using-aws-management-tools/
The post covers some of the techniques we use for configuration management at my company, many of which I helped design.
Thanks for giving it a read through!
Deep Archive is about half the cost of Glacier at $0.00099 per GB per month, but you sacrifice the option to get your data back in minutes, your only retrieval option is hours.
I work for a health care company so we hold onto patient data for years. There are plenty of reasons we might need to retrieve data from years ago, but few of them would have a time limit of less than several weeks. That makes Deep Archive a great fit for our long term data retention.
Setting it up is as simple as changing an existing life cycle transition to Deep Archive, or creating a new S3 Lifecycle transition to glacier
We put together a quick script to find the lifecycle transition rules in our S3 buckets that move data to Glacier already
$buckets = get-s3bucket;
# Iterate through buckets in the current account
foreach ($bucket in $buckets) {
write-host -foregroundcolor Green "Bucket: $($bucket.BucketName)";
# Get the lifecycle configuration for each bucket
$lifecycle = Get-S3LifecycleConfiguration -BucketName $bucket.BucketName;
# Print a warning if ther eare no lifecycles for this bucket
if(!$lifecycle) {
write-host -foregroundcolor Yellow "$($bucket.BucketName) has no life cycle policies";
} else {
# Iterate the transition rules in this lifecycle
foreach ($rule in $lifecycle.Rules) {
write-host -foregroundcolor Magenta "$($rule.Id) with prefix: $($rule.Filter.Lifecyclefilterpredicate.Prefix)";
# Print a warning if there are no transitions
if(!($rule.Transitions)) {
write-host -foregroundcolor Yellow "No lifecycle transitions";
}
# Iterate the transitions and print the rules
foreach ($transition in $rule.Transitions) {
if($transition.StorageClass -eq "GLACIER") {
$color = "Yellow";
} else {
$color = "White";
}
write-host -foregroundcolor $color "After $($transition.Days) transition to $($transition.StorageClass)";
}
}
}
}
To run this script you’ll need the AWS powershell tools, an IAM account setup, and a default region initialized.
When you run the script it will print out your current S3 buckets, the lifecycle rules, and the transitions in each of them, highlighting the transitions to Glacier in yellow.
Our solution for balancing this is a powershell snippet that graphs our RI commitment over time.
# Get RI entries from AWS console
$ri_entries = Get-EC2ReservedInstance -filter @(@{Name="state";Value="active"});
# Array to hold the relevant RI data
$ri_data = @();
# Calculate monthly cost for RIs
foreach ($ri_entry in $ri_entries) {
$ri = @{};
$hourly = $ri_entry.RecurringCharges.Amount;
$monthly = $hourly * 24 * 30 * $ri_entry.InstanceCount;
$ri.monthly = $monthly;
$ri.End = $ri_entry.End;
$ri_data += $ri;
}
# Three years into the future (maximum duration of RIs as of 1.22.2019)
$three_years_out = (get-date).addyears(3);
# Our current date iterator
$current = (get-date);
# Array to hold the commit by month
$monthly_commit = @();
# CSV file name to save output
$csv_name = "ri_commitment-$((get-date).tostring('ddMMyyyy')).csv";
# Remove the CSV if it already exists
if(test-path $csv_name) {
remove-item -force $csv_name;
}
# Insert CSV headers
"date,commitment" | out-file $csv_name -append -encoding ascii;
# Iterate from today to three years in the future
while($current -lt $three_years_out) {
# Find the sum of the RIs that are active on this date
# all RI data -> RIs that have expirations after current -> select the monthly measure -> get the sum -> select the sum
$commit = ($ri_data | ? {$_.End -gt $current} | % {$_.monthly} | measure -sum).sum;
# Build a row of the CSV
$output = "$($current),$($commit)";
# Print the output to standard out for quick review
write-host $output;
# Write out to the CSV for deeper analysis
$output | out-file $csv_name -append -encoding ascii;
# Increment to the next month and repeat
$current = $current.addmonths(1);
}
Ok, short’s not the right word. It’s a little lengthy, but at the end it kicks out a CSV in your working directory with months and your RI commit for them.
From there it’s easy to create a graph that shows your RI spend commit over time.
That gives you an idea of how much spend you’ve committed to, and for how long.
Rolling Websphere 