pester – Rolling Websphere

I recently wrote an intro to unit testing your powershell modules with Pester, and I wanted to give a walk through for our method of running these unit tests inside of a Docker for Windows container.

Before we get started, I’d like to acknowledge this post is obviously filled with trendy buzzwords (CICD, Docker, Config Management, *game of thrones, docker-compose, you get the picture). All of the components we’re going to talk through today add concrete value to our business, and we didn’t do any resume driven development.

Why?

Here’s a quick run through of our motivation for each of the pieces I’ll cover in this post.

Docker image for running unit tests
1. gives engineers a consistent way to run the unit tests. One your workstation you might need different versions of SDKs and tools, but a docker container lets you pin versions of things like the AWS Powershell tools
2. Makes all pathing consistent – you can setup your laptop anyway you lock, but the paths inside of the container are consistent
Docker-compose
1. Provides a way to customize unit test runs to a project
2. Provides a consistent way for engineers to map drives into the container
Code coverage metrics
1. At my company we don’t put too much stock in code coverage metrics, but they offer some context for how thorough an engineer has been with unit tests
2. We keep a loose goal of 60%
Unit test passing count
1. A failed unit test does not go to production. A failed unit test has a high chance of causing production outage

How!

The first step is to setup Docker Desktop for Windows. The biggest struggle I’ve seen people having getting docker running on Windows is getting virtualization enabled, so pay extra attention to that step.

Once you have Docker installed you’ll need to create an image you can use to run your unit tests, a script to execute them, and a docker-compose file. The whole structure will look like

/
- docker-compose.yml
- /pestertester
  - Dockerfile
  - Run-AllUnitTests.ps1

We call our image “pestertester” (I’m more proud of that name than I should be).

There are two files inside of the pestertester folder: a Dockerfile that defines the image, and a script called Run-AllUnitTests.ps1.

Here’s a simple example of the dockerfile. For more detail on how to write a dockerfile you should explore the dockerfile reference

FROM mcr.microsoft.com/windows/servercore
RUN "powershell Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force"
RUN "powershell Install-Module -Scope CurrentUser -Name AWSPowerShell -Force;"
COPY ./Run-AllUnitTests.ps1 c:/scripts/Run-AllUnitTests.ps1

All we need for these unit tests is the AWS Powershell Tools, and we install NuGet so we can use powershell’s Install-Module.

We played around with several different docker images before we picked mcr.microsoft.com/windows/servercore.

We moved away from any of the .NET containers because we didn’t need the dependencies they added, and they were very large
We moved away from nano server images because some of our powershell modules call functions outside of .NET core

Next we have the script Run-AllUnitTests.ps1. The main requirement for this script to work is that your tests be stored with this file structure

/ConfigModule.psm1
- /tests
  - /ConfigModule.tests.ps1
ConfigModule2.psm1
- /tests
  - /ConfigModule2.tests.ps1

The script isn’t too complicated

$results = @();
gci -recurse -include tests -directory | ? {$_.FullName -notlike "*dsc*"} | % {
    set-location $_.FullName;
    $tests = gci;
    foreach ($test in $tests) {
        $module = $test.Name.Replace("tests.ps1","psm1")
        $result = invoke-pester ".\$test" -CodeCoverage "..\$module" -passthru -quiet;
        $results += @{Module = $module;
                      Total = $result.TotalCount;
                      passed = $result.PassedCount;
                      failed = $result.FailedCount
                      codecoverage = [math]::round(($result.CodeCoverage.NumberOfCommandsExecuted / $result.CodeCoverage.NumberOfCommandsAnalyzed) * 100,2)
                    }
    }
}

foreach ($result in $results) {
    write-host -foregroundcolor Magenta "module: $($result['Module'])";
    write-host "Total tests: $($result['total'])";
    write-host -ForegroundColor Green "Passed tests: $($result['passed'])";
    if($result['failed'] -gt 0) {
        $color = "Red";
    } else {
        $color = "Green";
    }
    write-host -foregroundcolor $color "Failed tests: $($result['failed'])";
    if($result['codecoverage'] -gt 60) {
        $color = "Green";
    } elseif($result['codecoverage'] -gt 30) {
        $color = "Yellow";
    } else {
        $color = "Red";
    }
    write-host -ForegroundColor $color "CodeCoverage: $($result['codecoverage'])";
}

The script iterates through any subdirectories named “tests”, and executes the unit tests it finds there, running code coverage metrics for each module.

The last piece to tie all of this together is a docker-compose file. The docker compose file handles

Mapping the windows drives into the container
Executing the script that runs the unit tests

The docker-compose file is pretty straightforward too

version: '3.7'

services:
  pestertester:
    build: ./pestertester
    volumes:
      - c:\users\bolson\documents\github\dt-infra-citrix-management\ssm:c:\ssm
    stdin_open: true
    tty: true
    command: powershell "cd ssm;C:\scripts\Run-AllUnitTests.ps1"

Once you’ve got all of this setup, you can run your unit tests with

docker-compose run pestertester

One the container starts up you’ll see your test results

Experience

We’ve been running linux containers in production for a couple of years now, but we’re just starting to pilot windows containers. According to the documentation they’re not production ready yet

Docker is a full development platform for creating containerized apps, and Docker Desktop for Windows is the best way to get started with Docker on Windows.

Running our unit tests inside of windows containers has been a good way to get some experience with them without risking production impact.

A couple final thoughts

Windows containers are large, even server core and nano server are gigabytes.

The container we landed on is 11GB

If you need to run windows containers, and you can’t stick to .NET core and get onto nano server, you’re going to be stuck with pretty large images.

Start up times for windows containers will be a few minutes

Especially the first time on a machine while resources are getting loaded.

Versatile Pattern

This pattern of unit testing inside of a container is pretty versatile. You can use it with any unit testing framework, and any operating system you can run inside a container.

*no actual game of thrones references will be in this blog post

Pester is a unit testing framework for Powershell. There are some good tutorials for it on their github page, and a few other places, but I’d like to pull together some of the key motivating use cases I’ve found and a couple of the gotchas.

Let’s start with a very simple example.

This is the contents of a simple utility module named Util.psm1

function Get-Sum([int]$number1, [int]$number2) {
    $result = $number1 + $number2;
    write-host "Result is: $($result)";
    return $result;
}

And this is the content of a simple unit test file named UtilTest.ps1

Import-Module .\Util.psm1
Describe "Util Function Tests" {
    It "Get-Sum Adds two numbers" {
        Get-Sum 2 2 | Should be 4;
    }
}

We can run these tests using “Invoke-Pester .\UtilTest.ps1”.

And already there’s a gotcha here that wasn’t obvious to me from the examples online. Let’s say I change my function to say “Sum is:” instead of “Result is” and save the file. When I re-run my pester tests I still see “Result is:” printed out.

What’s also interesting is that the second run rook 122 ms, while the first took 407 ms.

It turns out both of these changes are results of the same fact – once the module you are testing is loaded into memory it will stay there until you Remove it. That means any changes you make trying to fix your unit tests won’t take effect until you’ve refreshed the module. The fix is simple

Import-Module .\Util.psm1
Describe "Util Function Tests" {
    It "Get-Sum Adds two numbers" {
        Get-Sum 2 2 | Should be 4;
    }
}
Remove-Module Util;

Removing the module after running your tests makes powershell pull a fresh copy into memory so you can see the changes.

The next gotcha is using the Mock keyword. Let’s say I want to hide the write-host output in my function so it doesn’t clutter up my unit tests. The obvious way is to use the “Mock” keyword to create a new version of write-host that doesn’t actually write anything. My first attempt looked like this

Import-Module .\Util.psm1
Describe "Util Function Tests" {
    It "Get-Sum Adds two numbers" {
        Mock write-host;
        Get-Sum 2 2 | Should be 4;
    }
}
Remove-Module Util;

But I still see the write-host output in my unit test results.

It turns out the reason is that the Mock keyword creates mock objects in the current scope, instead of in scope for the module being tested. There are two ways of fixing this. One is the InModuleScope, or the ModuleName parameter on the Mock object. Here’s an example of the first option

Import-Module .\Util.psm1

InModuleScope Util {
    Describe "Util Function Tests" {
        It "Get-Sum Adds two numbers" {
            Mock write-host;
            Get-Sum 2 2 | Should be 4;
        }
    }
}
Remove-Module Util;

And just like that the output goes away!

Rolling Websphere

Tag: pester

Docker Windows container for Pester Tests