PlayNicely.Executor 1.3.2-beta-507

Play Nicely - Executor

The Play Nicely Executor project extends the PlayNicely.Projects package to support execution of programs, against test case projects. The execution occurs within a pre-defined, side-effect free, environment. It then collects program output so that test assertions can be made.

The Executor package, provides a fluent interface to define an ITestEnvironment through a builder, with which, some programs can be defined as uninstalled, other programs as required. A test environment can also define what NuGet package sources are available and what packages are mapped to those sources, so that BDD projects can test new functionality. If a test case project is specified, the file system of that project will be 'materialized' and set as the working directory for the executed process.

How It Works

The main types are the ITestEnvironment, TestEnvironmentBuilder, and ITestEnvironmentRunner. An ITestEnvironment ensures a side-effect free environment, within which, any testing can run. You create an ITestEnvironment using the fluent interface of TestEnvironmentBuilder. With this class you define the desired working directory and execution context for your ITestEnvironmentRunner.

Tests run using the ITestEnvironmentRunner, this package defines a basic ProcessRunner which runs an executable with arguments, much like a command line, and returns a basic ExecutionResult. The base result includes whether the command succeeded and the stdout and stderr streams. The DotNetRunner class, defined in PlayNicely.Executor.DotNet executes a dotnet subcommand, with arguments, and returns a ExecutionResult<DotNetExecutionContext>. This includes detailed information about the dotnet command results, build context, targets ran, projects built, lists of errors, etc.

If you have a specific process you'd like to support, you can define a concrete implementation derived from ITestEnvironmentRunner.

Getting Started

This getting started uses the ProcessRunner to execute a dotnet build process on a pre-defined test case project, within a side-effect free test environment. The purpose of this getting started is to demonstrate how to set up a test environment, execute a runner and assert the result.

ℹ️ This is only an example
I'm using ProcessRunner here with dotnet.exe to illustrate getting started. The PlayNicely.Executor.DotNet package provides a specific implementation for dotnet that includes important context information after execution. If you are running dotnet tests, we recommend using that runner instead.

This getting started is code-first, in a typical configuration you would likely use the IDE to define test case projects and SpecFlow (or some other BDD framework) to define the environment.

Define the test case project

Let's create a scenario to test for a failing build, because the .NET target framework is invalid. First, define the test case project and file system.

var testCaseProject = new TestCaseProject("my-failing-project");
var projectFile = testCaseProject.Root.AddFile("proj.csproj");

testCaseProject.ProjectFile = projectFile;

using(var writer = new StreamWriter(projectFile.OpenWriteStream())
{
    writer.WriteLine("<Project Sdk=\"Microsoft.NET.Sdk\">");
    writer.WriteLine("  <PropertyGroup>");
    // Note invalid target framework...
    writer.WriteLine("    <TargetFramework>my-net9.0</TargetFramework>");
    writer.WriteLine("  </PropertyGroup>");
    writer.WriteLine("</Project>");
}

Define the environment

Use the TestEnvironmentBuilder to specify:

• Command Management
  • RequiredCommands
    Any command that must be available in the test environment. Using this method ensures commands can always be executed, by finding them in the current $PATH, and creating a temporary bin directory with symbolic links to the actual executables. This bin directory is prepended to the $PATH in the subsequent environment. On Windows this is less likely to be required, but on Linux, most commands are in shared directories like /usr/bin or /usr/local/bin, if something is excluded (see next bullet point), by removing one of these paths, it is likely to exclude a lot of other commands, RequiredCommands ensures the essential commands can still be executed from the $PATH.
  • ExcludeCommandFromPath
    It is often neccesary, in a negative test case, to assume a program is not installed. This method locates a command on the current $PATH and removes any directories where it is found. The result of this, when running in the ITestEnvironment is that any attempt to run an excluded command will fail (because it isn't found on the $PATH).
• Package Management

  • AddPackageSource
    Often, the usage of this package is to test another NuGet package project. So that the test environment can run release tests using the under development version of a package, this method allows overriding NuGet.Config in the test environment.

    ℹ️ If you specify a relative path for the source location, it is relative to your assembly's binary location.

  • ClearPackageSources
    Allows the list of package sources to be cleared so that machine wide sources can be excluded during testing.

  • MapPackagesToSource
    Map packages with a package name (pattern) to a specific source. So that you can test local packages rather than the most recent one published to nuget.org. This is important, if you are making a change to a package you want to test those changes locally.

  • PackageSourceFallback
    After mapping patterns to particular package sources, you need to set an all encompassing fallback package source. You can use this method to do that if you know which source, typically nuget.org. If you'd rather use a best default, see the next method.

  • UseBestPackageSourceFallback
    If you would rather just default the fallback package source, use this method. It will use the following logic to determine a fallback source.

    • use source named 'nuget.org'
    • use the first source that is https?:// and is using the domain 'nuget.org'
    • use the first source that is https:// based
    • use the first source that is http:// based
    • use any available source
• SetProject
  The target project for this test environment, the project's file system will be materialized and set as the working directory for the process.

Example for clarity

Continuing context from here. We need to ensure dotnet is always available, and specify the test case target project.

var builder = new TestEnvironmentBuilder();
var testEnv = await builder.RequiredCommands("dotnet")
                           .SetProject(testCaseProject)
                           .BuildAsync();

Run the test

With the environment built, simply construct the runner and assert the result (which we expect to fail).

var runner = new ProcessRunner("dotnet", "build"); // Don't need to specify project path,
                                                   // process working dir is root of project
var testResult = await runner.ExecuteAsync(testEnv);

Assert.That(testResult.Succeeded, Is.False);

Why?

This project came about to support the use of Node.js packages within .NET projects in a .NET first way. You can achieve the integration of Node.js tools using plugins or other tooling. The problem with this (using plugins) is, these can often become out of date or stale. Most of the Node.js packages are developed by an active community, so accessing the latest npm packages directly makes the most sense (we get latest features and security updates).