skUnit 0.31.0-beta

skUnit

skUnit is a testing tool for SemanticKernel units, such as plugin functions and kernels.

For example, you can use skUnit to test a GetSentiment function that analyzes a text and returns its sentiment, such as "Happy" or "Sad". You can write different scenarios to check how the function behaves with various inputs, such as:

## PARAMETER input
Such a beautiful day it is

## ANSWER Equals
Happy

This scenario verifies that the function returns "Happy" when the input is "Such a beautiful day it is".

This is an Invocation Scenario, which tests a single function call. You can also write Chat Scenarios, which test a sequence of interactions between the user and the SemanticKernel.

skUnit offers many features to help you write more complex and flexible scenarios. In this section, we will show you some of them with an example.

Suppose you have a function called GetSentiment that takes two parameters and returns a sentence describing the sentiment of the text:

Parameters:

• input: the text to analyze
• options: the possible sentiment values, such as happy, angry, or sad

Returns: a sentence like "The sentiment is happy" or "The sentiment of this text is sad".

Here is a scenario that tests this function:

# SCENARIO GetSentimentHappy

## PARAMETER input
Such a beautiful day it is

## PARAMETER options
happy, angry

## ANSWER SemanticSimilar
The sentiment is happy

The most interesting part of this scenario is:

## ANSWER SemanticSimilar
The sentiment is happy

This line specifies the expected output of the function and how to compare it with the actual output. In this case, the output should be semantically similar to "The sentiment is happy". This means that the output can have different words or syntax, but the meaning should be the same.

This is a powerful feature of skUnit scenarios, as it allows you to use OpenAI itself to perform semantic comparisons.

You can also write this assertion in another way:

## ANSWER
The sentiment of the sentence is happy

## CHECK SemanticSimilar
The sentiment is happy

In this style, the expected answer is just a reminder and not used for comparison; and then a ## CHECK SemanticSimilar is used to explicitly perform the assertion.

However, SemanticSimilar is not the only assertion method. There are many more assertion checks available (like SemanticCondition, Equals).

You can see the full list of CHECK statements here: CHECK Statement spec.

Scenarios are Valid Markdowns

One of the benefits of skUnit scenarios is that they are valid Markdown files, which makes them very readable and easy to edit.

skUnit scenarios are valid Markdown files, which makes them very readable and easy to edit.

For example, you can see how clear and simple this scenario is: Chatting about Eiffel height

Executing a Test Using a Scenario

Executing tests is a straightforward process. You have the flexibility to utilize any preferred test frameworks such as xUnit, nUnit, or MSTest. With just two lines of code, you can load and run a test:

var scenarios = InvocationScenario.LoadFromText(scenario);
await SemanticKernelAssert.CheckScenarioAsync(Kernel, scenarios);

The standout feature of skUnit is its detailed test output. Here's an example:

# SCENARIO GetSentimentHappy_Fail

## PARAMETER input
You are such a bastard, Fuck off!

## PARAMETER options
happy, angry

## EXPECTED ANSWER
The sentiment is happy.

## ACTUAL ANSWER
angry

## ANSWER SemanticSimilar
The sentiment is happy
Exception as EXPECTED:
The two texts are not semantically equivalent. The first text expresses anger, while the second text expresses happiness.

As demonstrated, when a SemanticSimilar check fails, it provides a semantic explanation for the failure. This feature proves to be incredibly useful during debugging.

Here's another example of an executing The Chatting about Eiffel height test:

Documents

To better understand skUnit, Check these documents:

• Invocation Scenario Spec: The details of writing an InvocationScenario.
• Chat Scenario Spec: The details of writing an ChatScenario.
• CHECK Statement Spec: The various CHECK statements that you can use for assertion.

Requirements

• .NET 7.0 or higher
• An OpenAI API key

Installation

You can easily add skUnit to your project as it is available as a NuGet package. To install it, execute the following command in your terminal:

dotnet add package skUnit

Afterwards, you'll need to instantiate the SemanticKernelAssert class in your test constructor. This requires passing your OpenAI subscription details as parameters:

public class MyTest
{
  SemanticKernelAssert SemanticKernelAssert { get; set; }
  MyTest(ITestOutputHelper output)
  {
    SemanticKernelAssert = new SemanticKernelAssert(_deploymentName, _endpoint, _apiKey, message => output.WriteLine(message));
  }

  [Fact]
  MyFunctionShouldWork()
  {
    var scenarios = await InvocationScenario.LoadFromResourceAsync(scenario);
    await SemanticKernelAssert.CheckScenarioAsync(Kernel, scenarios);
  }
}

And that's all there is to it! 😊