Xunit.Extensions.Ordering 1.4.0

Xunit.Extensions.Ordering

Xunit extension that provides full support for ordering at all levels - test collections, test classes and test cases.The common scenarion where ordering is useful is integration testing.

Extension provides full-featured AssemblyFixture implementation with same functionality as class and collection fixtures (including IMessageSink injection, support for IAsyncLifetime).

Supports: .NET Core 1.x, .NET Core 2.x. and may work in *.NET 4.5.2+

Nuget: https://www.nuget.org/packages/Xunit.Extensions.Ordering/

Table of contents

1. Test cases ordering
   1. Setup ordering
   2. Ordering classes and cases
   3. Ordering classes in collection
   4. Ordering collections
   5. Mixing test classes with and without explicit collection assignement
   6. Checking continuity and duplicates
   7. Notes
2. AssemblyFixture
   1. Setup Fixture
   2. Basic usage
   3. Multiple assembly fixtures
   4. IAsyncLifetime
   5. IAssemblyFixture<TFixture>
   6. Notes about AssemblyFixture implementation

Test cases ordering

Setup ordering

Add AssemblyInfo.cs with only following lines of code

using Xunit;
//Optional
[assembly: CollectionBehavior(DisableTestParallelization = true)]
//Optional
[assembly: TestCaseOrderer("Xunit.Extensions.Ordering.TestCaseOrderer", "Xunit.Extensions.Ordering")]
//Optional
[assembly: TestCollectionOrderer("Xunit.Extensions.Ordering.CollectionOrderer", "Xunit.Extensions.Ordering")]

Ordering classes and cases

Add Order attribute to test classes and methods. Tests are executed in ascending order. If no Order attribute is specified default 0 is assigned. Multiple Order attributes can have same value. Their execution order is in this case deterministic but unpredictible.

[Order(1)]
public class TC2
{
	[Fact, Order(2)]
	public void M1() { /* ... */ }

	[Fact, Order(3)]
	public void M2() { /* ... */ }

	[Fact, Order(1)]
	public void M3() { /* ... */ }
}

Ordering classes in collection

You can order test classes in collections by adding Order attribute but you have to use patched test framework by adding following lines to AssemblyInfo.cs

using Xunit;

[assembly: TestFramework("Xunit.Extensions.Ordering.TestFramework", "Xunit.Extensions.Ordering")]

Order collections like this

[CollectionDefinition("C1")]
public class Collection1 { }

[Collection("C1"), Order(2)]
public class TC3
{
	[Fact, Order(1)]
	public void M1() { /* 3 */ }

	[Fact, Order(2)]
	public void M2() { /* 4 */ }
}

[Collection("C1"), Order(1)]
public partial class TC5
{
	[Fact, Order(2)]
	public void M1() { /* 2 */ }

	[Fact, Order(1)]
	public void M2() { /* 1 */ }

}

Ordering collections

You can order test collections by adding Order attribute too definition collection class

[CollectionDefinition("C1"), Order(3)]
public class Collection3 { }

[CollectionDefinition("C2"), Order(1)]
public class Collection3 { }

Mixing test classes with and without explicit collection assignement

Test classes without explicitely assigned collection are collections implicitely in Xunit (collection per class). If you mix both types of collections they are on the same level and Order is applied following this logic.

[CollectionDefinition("C1"), Order(3)]
public class Collection3 { }

[CollectionDefinition("C2"), Order(1)]
public class Collection3 { }

[Order(2)]
public class TC2
{
	[Fact]
	public void M1() { /* 4 */ }
}

[Collection("C1")]
public class TC3
{
	[Fact]
	public void M1() { /* 5 */ }
}

[Collection("C2"), Order(2)]
public partial class TC5
{
	[Fact]
	public void M1() { /* 3 */ }
}

[Collection("C2"), Order(1)]
public partial class TC5
{
	[Fact, Order(2)]
	public void M1() { /* 2 */ }

	[Fact, Order(1)]
	public void M2() { /* 1 */ }
}

Checking continuity and duplicates

You can enable warning messages about continuity and duplicates of order indexes.

1. Create xnuit.runner.json file in root of your test project

{
	"$schema": "https://xunit.github.io/schema/current/xunit.runner.schema.json",
	"diagnosticMessages": true
}

2. Set "Copy to output directory" for this file to "Copy if newer"
3. In the Output window choose "Tests" option in the "Show output from" dropdown or just run dotnet test from Package Manager Console
4. You'll start getting warnings like

Missing test collection order sequence from '4' to '39'.
Missing test case order '1' in test class 'Xunit.Extensions.Ordering.Tests.TC6'.
Missing test classes order sequence from '3' to '29' for collection 'C1'.
Missing test case order sequence from '2' to '19' in test class 'Xunit.Extensions.Ordering.Tests.TC5'.

Notes

There is no guarantee for Theory method execution order what is expected behavior.

[Theory, Order(4)]
[InlineData(15)]
[InlineData(16)]
[InlineData(17)]
public void Method(int expectedOrder) { Assert.Equal(expectedOrder, Counter.Next()); }

AssemblyFixture

Assembly fixtures are instantiated ones per test run. Assembly fixtures fully support IAsyncLifetime interface, injection of IMessageSink.

Setup fixture

Add AssemblyInfo.cs with only following lines of code

using Xunit;
using Xunit.Extensions.Ordering;

[assembly: AssemblyFixture(typeof(AsmFixture1))]
[assembly: AssemblyFixture(typeof(AsmFixture2))]

Basic usage

public class TC
{
	private readonly AsmFixture1 _fixture;

	public TC(AsmFixture1 fixture)
	{
		_fixture = fixture;
	}
}

Multiple assembly fixtures

public class TC
{
	private readonly AsmFixture1 _fixture1;
	private readonly AsmFixture2 _fixture2;
	private readonly ITestOutputHelper _output;

	public TC(AsmFixture1 fixture1, ITestOutputHelper output, AsmFixture2 fixture2)
	{
		_fixture1 = fixture1;
		_fixture2 = fixture2;
		_output = output;
	}
}

IAsyncLifetime

public class AsmFixture : IAsyncLifetime
{
	public IMessageSink MesssageSink { get; }
	public bool Initialized { get; private set; } = false;

	public AsmFixture(IMessageSink messsageSink)
	{
		MesssageSink = messsageSink;
	}

	public async Task InitializeAsync()
	{
		await Task.Run(() => { Initialized = true; });
	}

	public async Task DisposeAsync()
	{
		await Task.Run(
			() => MesssageSink.OnMessage(
				new DiagnosticMessage("Disposed async.")));
	}
}

IAssemblyFixture<TFixture>

You can use IAssemblyFixture<TFixture> as marker interface. Assembly fixtures are currently injected to constructor regardless of this interface. I will add option for smart resolving and instantiation of assembly fixtures only required by current test run.

public class TC : 
	IAssemblyFixture<AsmFixture1>,
	IAssemblyFixture<AsmFixture2>
{
	private readonly AsmFixture1 _fixture1;
	private readonly AsmFixture2 _fixture2;

	public TC(AsmFixture1 fixture1, AsmFixture2 fixture2)
	{
		_fixture1 = fixture1;
		_fixture2 = fixture2;
	}
}

Notes about AssemblyFixture

I don't split this functionality into two packages bcs. I need to rewrite TestFramework for ordering puposes. AssemblyFixtures are often used side by side with ordering.

Kick started by Xunit example by Brad Wilson. I've presered his original comments where it was applicable.