Category: docker

Chef Server and the risks of depending on the open source community

On By Brian OlsonIn chef, docker, open source, toolsLeave a comment

As a DevOps engineer I’ve played around with a number of different automation tools. Some professionally, some for a blog post, and some just because I was curious about them. When most companies are evaluating automation tools they look at some common attributes

  1. How much will it cost?
  2. Does it support the platforms we’re on?
  3. Is it a good fit for my teams skills?
  4. How long will it take to setup and see some value add?

And then there run down the list of common tools like Docker, Ansible, Chef, Puppet, Powershell DSC, and on until they find one that fits their needs will enough. But I’d argue there’s another aspect to most of these tools that’s harder to quantify, but more impactful to the long term success of a project or tool: how excited is the open source community about the tool.

Most of these tools have at least some level of dependency on open source development, whether it’s the Moby Linux vm behind running Linux containers on Windows machines, or the chef cookbooks in your Berksfile, you’re probably using something that’s open source. The effect that has on your project success can be subtle.

I’ve had two projects at work I’d like to walk you through: migrating a Django app into Docker, and adding AWS’s boto3 to a server using chef.

When I started on the Dockerfile for the Django app I googled “Django app in docker” and immediately found several good examples with notes, explanations, and a simple demo app. I could pick my Django and python versions using image tags, and I had my app starting in a container in about an hour (after that I spent a day hunting down undocumented environment variables, but that’s on us, not docker).

Overall it was a smooth process, there was rich documentation, and every question about tooling had already been answered on forums.

Compare that to trying to add a python package to a server using chef. I started off fighting with chef test kitchen. It took my an hour or so to figure out all of the variables I needed to create a server with the ec2 kitchen driver (env vars, values swapped out in my kitchen.yml file, drivers I needed to download, etc.)

Once I had an instance created in AWS I had to find an example of installing a python package. After 20 or so minutes of googling I landed on Poise python, which mostly works, but hasn’t been updated in close to a year.

The package I needed was psycopg2, and I got it added to my cookbook pretty easily, but then I tried to rerun kitchen converge, and I got an error from pretty deep inside of poise python. I’ve always believed part of the unwritten contract you agree to with open source is digging into bugs when you find them, so I dove in and eventually found the problem in here doc that creates a python script to install pip modules.

Looking at the heredoc gave me a few ideas for tweaking my chef recipe to use poise python differently and avoid the error, and after experimenting for a while I found a combination of version pinning that worked and I could converge my node.

Not long after that I a commit and PR ready to add the python module to the new node and I was done.

So what’s the difference here? The work I was doing was pretty similar in both situations, I needed to update packages for a python application, but the docker experience was smooth, there were lots of examples, and it only took a few hours total. The chef experience was painful, I spent time digging through source code, debugging my own problems, and had trouble finding examples.

To be clear I’m not promoting Docker over Chef. They’re different tools that solve similar problems. I’m trying to point out that community support has moved away from Chef and the open source Chef cookbooks over the last 3–5 years. That transition makes the tool much harder to use because there are fewer people to contribute and help solve problems.

And the scary thing is that the exact same thing could happen to Docker, leaving all of the companies containerizing their applications high and dry with less free community support than they have now. The open source world is enamored with Docker today, but they may not be tomorrow. So before you commit to an open source tool for the cost or other reasons above, I’d encourage you to ask these questions too

  1. Am I willing to contribute back to this open source project to solve my own problems?
  2. Is the open source community moving towards, or away from this project?
  3. How bad is it for my org if we start to find ourselves with less open source support for this?
  4. Am I willing to pay for enterprise support, or hire an open source community member if it comes to it?

Loving Docker as a Windows guy: August 2019

On By Brian OlsonIn docker, toolsLeave a comment
Over the last few years I’ve slowly made the transition from “docker detractor” through “docker accepter” all the way to “docker enthusiast” where I am now. Docker isn’t the only container tool out there, but it’s certainly one of the most popular. It gives a spin on application management and deployment that’s honestly pretty refreshing.

At the same time, I’ve been a windows enthusiast my whole life. I’m not opposed to Ubuntu, RHEL, or OSX, and I’m not a stranger to grep, find, and ls, but I’ll defend the use of Windows Server, “where-object” and “get-childitem” any day.
The Docker experience on windows has evolved a ton in the last few years. If you haven’t downloaded Docker Desktop, I would highly recommend grabbing it and playing around (even just to win a game of “have you tried?” at work).
In this post I’ll walk through some of my surprises (pleasant and unpleasant) with using Docker on windows for the last few years.

Volume Mapping is useful, but little tricky

If you didn’t know docker lets you mount directories from your host OS into your container with the docker run –volume option
This works between windows and linux containers, and windows and windows containers, and can be really useful for running linux utilities to modify files on your windows machine, like using a current version of openssl to update certificates

What’s weird about it?

Anytime you’re moving files between windows and linux, you’re going to encounter some weirdness. Things at the top of the list are
  • Files created on your windows machine show up in linux with “-rwxr-xr-x”. This can be a little confusing if you author a bash script on your laptop, then submit the Dockerfile to a build tool like CircleCI, and realize it doesn’t have execute permissions
  • Default line endings will drive you crazy
  • Support for specify the path on your host is different in Docker and docker-compose
    • Docker supports relative pathing, because you can execute “(pwd).path” inside of the string you pass to -v
    • Docker-compose supports relative pathing inside of your workspace with the normal “./”

Alpine will make you hate Windows Server Core

Alpine containers are delightfully tiny. Think of them like cheese burger sliders, or mini bagels, or minimuffins (you get the picture). The base alpine container is single digit MB (that’s not an exaggeration). Obviously as you add packages and tools to it, it starts to grow, but that’s a great starting point.
Compare that to Windows Server Core that starts off at just over 5 GB. And that only grows when you start adding packages and tools. By the time we have a working windows container it’s often in the tens of GB.
Partly because of the size windows server core containers take a long time to run some basic operations, like extracting, starting, and stopping. It can be frustrating compared to Alpine that can be completely downloaded and up and going in a matter of seconds.

Windows Server Nano is better

Microsoft Nano server is a smaller version of Windows Server and starts off at just under 400 MB which makes it smaller, but that shrinking comes with some caveats like it only runs .NET core and not full .NET.
That makes the case for Nano Server a little more obscure. If you’re committed to some specific windows features in .NET core that aren’t available in linux, it makes sense. But most of .NET core runs on linux now, so think about migrating to linux.

Microsoft’s Image Tagging and Versioning Confuses Me

While I was writing this post, I tried 3 or 4 different image tags from docker hub before I found the right combination of architecture and OS version that would work on my laptop.
Understanding that error comes down to understanding what docker does at it’s core. It is not a virtualization tool, your containers share your host kernel, so you obviously need to have a kernel that’s compatible with the image you’re trying to run. It can still be frustrating to get the first container running.
I’ve also spent a lot of time trying to find the right image repo on docker hub. Searching some terms that seem logical to me like “windows” or “windows 2019” don’t always return the images you’d expect. I haven’t found a good central listing of “all the windows containers from microsoft”.

Docker Desktop’s Moby Linux VM is smooth

When you run a linux container on windows Docker Desktop is using Microsoft HyperV behind the scenes to create a small VM you can use to run linux containers.
The first time I found that out, it gave me some heart palpitations. “A container inside of a VM running on my windows laptop? The resource drain will be astronomical! The performance will be subterranean! The compatibility will be atrocious!”
But after a few weeks of using it, I calmed down quite a bit. We’ve been running docker in production at work for a close to 2 years, and I have yet to see any compatibility issues pulling and running containers onto my laptop. Most of the development happens on OSX or actual linux machines, but the Moby VM works quite well.
This is going to change with the Windows Subsystem for linux 2, and I think that’s the right move. Microsoft is making Linux tooling and development first class citizens on Windows. But the linux VM running inside of HyperV is stable enough I won’t be an early adopter.

Networking is cool and a little confusing

It’s pretty straightforward to expose a port on the container to your host
And that works really smoothly, especially on Windows containers.
On linux containers the VM running your container is technically behind a HyperV switch, so some docker networking modes like “host” and “bridged” don’t work like you’d expect them to. This is fine if you’re running a single container, but when you want to create a group of containers with docker-compose you have to take it into consideration.
My recommendation is to keep things simple and use docker-compose as it was meant to. That is, let docker-compose manage the DNS and networking behind the scenes, and expose as few ports into your container services as you can.

Editing text files in windows containers is really hard, apparently?

If you’ve spent any time in linux you’re probably competent with at least one command line text editor (nano, vi, etc). But apparently none of those are readily available in most versions of windows containers. Searching the internet for “editing text files in powershell” leads me to a bunch of articles about using “set-content” or using powershell to open notepad.
Those approaches have their uses, but what if I just want to edit a config file in a windows container? Apparently my only option is to write a regex, or map the volume to my host. Weird.

Using Docker as an Admin tool on Windows

On By Brian OlsonIn beginner, opensslLeave a comment
Have you ever tried to download openssl on Windows? You need to convert a cert, or just do some crypto work, so you google “openssl windows” and find the source forge entry. After a few minutes of scrolling around confused you finally accept that the page doesn’t have a release more recent than several years ago.

So you go back to google and click on the link for openssl.org, and realize that they don’t distribute any binaries at all (windows or otherwise).
You scroll a few entries further down, still looking for an executable or guide to get openssl on windows, and you click on a promising article heading. Perusing it tells you that it’s actually just a guide for Cygwin (and it would work, but then you have Cygwin sitting on your machine, and you’ll probably never use it again). You think to yourself, “There has to be an executable somewhere.”
Next you jump to page 2 of the google results (personally it’s the first time I’ve jumped to that page two in years) and scrolling you find more of the same. Linux fanatics using Cygwin, source code you could compile yourself, and obscure religious wars like schannel vs every other cryptography provider.
All you really want is to go from a .pfx to a .pem, and you’re running in circles looking for the most popular tool in the world to do it.
Enter Docker.
At work a number of our services are deployed on Docker, so I already have Docker desktop installed, and it’s often in Linux container mode on my workstation, so it was only a couple commands to get into openssl in an alpine container
Here are my commands for reference:
PS C:\Users\bolson\Documents> docker run -v "$((pwd).path)/keys:/keys" -it alpine
/ # cd keys/
/keys # ls -l | grep corp.pem
-rwxr-xr-x    1 root     root          1692 Jul  6 17:09 corp.pem
/keys # apk add openssl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/1) Installing openssl (1.1.1c-r0)
Executing busybox-1.30.1-r2.trigger
OK: 6 MiB in 15 packages
/keys # openssl
OpenSSL>

I realize there are plenty of other options for getting a Linux interpreter up and running on Windows. You could grab virtual box and a ubuntu ISO, you could open a Cloud9 environment on AWS and get to an Amazon Linux instance there, you could use the Windows Subsystem for Linux, you could dual boot your windows laptop with some linux distro, and the list could go on.

Those approaches are fine and would all work but they either take time, cost money, or are focused on one specific scenario, and wouldn’t have much utility outside of getting your into openssl to convert your cert. If I realize that one of the certs I need to convert is a jks store instead of a .pfx, I can flip over to a docker image with the java keytool installed pretty easily

Cleanup is easy with a few powershell commands

$containers = docker ps -a
foreach ($container in $containers) {$id = ($container -split "[ ]+")[0];docker rm $id}

$images = docker images;
foreach ($image in $images) {$id = ($image -split "[ ]+")[2];docker rmi $id}

And that’s why, as a windows user, I love Docker. You get simple, easy access to Linux environments for utilities and it’s straightforward to map directories on your windows machines into the Linux containers.

Nowadays you can use the Windows Subsystem for Linux for easy command line SSH access from Windows, but before that went GA on Windows 10 I used Docker for an easy SSH client (I know that plink exists, so this time you can accuse me of forcing Docker to be a solution).

You can create a simple Dockerfile that adds open ssh to an alpine container like so

FROM alpine
RUN apk add openssh-client

And then run it with

docker build -t ssh .
docker run -v "$($env:userprofile)/documents/keys:/keys" -it ssh sh

And you’re up and runing with an SSH client. Simple!

Again, there are other ways of accomplishing all of these tasks. But if you’re organization is investing in Docker using it for a few simple management tasks can give you some familiarity with the mechanics, and make it easier for you to support on all kinds of development platforms.

Docker Windows container for Pester Tests

On By Brian OlsonIn docker, scripting, unit testingLeave a comment
I recently wrote an intro to unit testing your powershell modules with Pester, and I wanted to give a walk through for our method of running these unit tests inside of a Docker for Windows container.

Before we get started, I’d like to acknowledge this post is obviously filled with trendy buzzwords (CICD, Docker, Config Management, *game of thrones, docker-compose, you get the picture). All of the components we’re going to talk through today add concrete value to our business, and we didn’t do any resume driven development.

Why?

Here’s a quick run through of our motivation for each of the pieces I’ll cover in this post.
  1. Docker image for running unit tests 
    1. gives engineers a consistent way to run the unit tests. One your workstation you might need different versions of SDKs and tools, but a docker container lets you pin versions of things like the AWS Powershell tools
    2. Makes all pathing consistent – you can setup your laptop anyway you lock, but the paths inside of the container are consistent
  2. Docker-compose
    1. Provides a way to customize unit test runs to a project
    2. Provides a consistent way for engineers to map drives into the container
  3. Code coverage metrics
    1. At my company we don’t put too much stock in code coverage metrics, but they offer some context for how thorough an engineer has been with unit tests
    2. We keep a loose goal of 60%
  4. Unit test passing count
    1. A failed unit test does not go to production. A failed unit test has a high chance of causing production outage

How!

The first step is to setup Docker Desktop for Windows. The biggest struggle I’ve seen people having getting docker running on Windows is getting virtualization enabled, so pay extra attention to that step.
Once you have Docker installed you’ll need to create an image you can use to run your unit tests, a script to execute them, and a docker-compose file. The whole structure will look like

  • /
    • docker-compose.yml
    • /pestertester
      • Dockerfile
      • Run-AllUnitTests.ps1

We call our image “pestertester” (I’m more proud of that name than I should be).

There are two files inside of the pestertester folder: a Dockerfile that defines the image, and a script called Run-AllUnitTests.ps1.
Here’s a simple example of the dockerfile. For more detail on how to write a dockerfile you should explore the dockerfile reference

FROM mcr.microsoft.com/windows/servercore
RUN "powershell Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force"
RUN "powershell Install-Module -Scope CurrentUser -Name AWSPowerShell -Force;"
COPY ./Run-AllUnitTests.ps1 c:/scripts/Run-AllUnitTests.ps1

All we need for these unit tests is the AWS Powershell Tools, and we install NuGet so we can use powershell’s Install-Module.

We played around with several different docker images before we picked mcr.microsoft.com/windows/servercore.

  1. We moved away from any of the .NET containers because we didn’t need the dependencies they added, and they were very large
  2. We moved away from nano server images because some of our powershell modules call functions outside of .NET core
Next we have the script Run-AllUnitTests.ps1. The main requirement for this script to work is that your tests be stored with this file structure
  • /ConfigModule.psm1
    • /tests
      • /ConfigModule.tests.ps1
  • ConfigModule2.psm1
    • /tests
      • /ConfigModule2.tests.ps1
The script isn’t too complicated
$results = @();
gci -recurse -include tests -directory | ? {$_.FullName -notlike "*dsc*"} | % {
    set-location $_.FullName;
    $tests = gci;
    foreach ($test in $tests) {
        $module = $test.Name.Replace("tests.ps1","psm1")
        $result = invoke-pester ".\$test" -CodeCoverage "..\$module" -passthru -quiet;
        $results += @{Module = $module;
                      Total = $result.TotalCount;
                      passed = $result.PassedCount;
                      failed = $result.FailedCount
                      codecoverage = [math]::round(($result.CodeCoverage.NumberOfCommandsExecuted / $result.CodeCoverage.NumberOfCommandsAnalyzed) * 100,2)
                    }
    }
}

foreach ($result in $results) {
    write-host -foregroundcolor Magenta "module: $($result['Module'])";
    write-host "Total tests: $($result['total'])";
    write-host -ForegroundColor Green "Passed tests: $($result['passed'])";
    if($result['failed'] -gt 0) {
        $color = "Red";
    } else {
        $color = "Green";
    }
    write-host -foregroundcolor $color "Failed tests: $($result['failed'])";
    if($result['codecoverage'] -gt 60) {
        $color = "Green";
    } elseif($result['codecoverage'] -gt 30) {
        $color = "Yellow";
    } else {
        $color = "Red";
    }
    write-host -ForegroundColor $color "CodeCoverage: $($result['codecoverage'])";
}

The script iterates through any subdirectories named “tests”, and executes the unit tests it finds there, running code coverage metrics for each module.

The last piece to tie all of this together is a docker-compose file. The docker compose file handles

  1. Mapping the windows drives into the container
  2. Executing the script that runs the unit tests
The docker-compose file is pretty straightforward too
version: '3.7'

services:
  pestertester:
    build: ./pestertester
    volumes:
      - c:\users\bolson\documents\github\dt-infra-citrix-management\ssm:c:\ssm
    stdin_open: true
    tty: true
    command: powershell "cd ssm;C:\scripts\Run-AllUnitTests.ps1"

Once you’ve got all of this setup, you can run your unit tests with 

docker-compose run pestertester

One the container starts up you’ll see your test results

Experience

We’ve been running linux containers in production for a couple of years now, but we’re just starting to pilot windows containers. According to the documentation they’re not production ready yet

Docker is a full development platform for creating containerized apps, and Docker Desktop for Windows is the best way to get started with Docker on Windows.

Running our unit tests inside of windows containers has been a good way to get some experience with them without risking production impact.

A couple final thoughts

Windows containers are large, even server core and nano server are gigabytes.

The container we landed on is 11GB

If you need to run windows containers, and you can’t stick to .NET core and get onto nano server, you’re going to be stuck with pretty large images.

Start up times for windows containers will be a few minutes

Especially the first time on a machine while resources are getting loaded.

Versatile Pattern

This pattern of unit testing inside of a container is pretty versatile. You can use it with any unit testing framework, and any operating system you can run inside a container.

*no actual game of thrones references will be in this blog post