AssemblyAI.SemanticKernel 1.0.2-alpha

<img src="https://github.com/AssemblyAI/assemblyai-python-sdk/blob/master/assemblyai.png?raw=true" width="500" alt="AssemblyAI logo"/>

AssemblyAI integration for Semantic Kernel

Transcribe audio using AssemblyAI with Semantic Kernel plugins.

Get started

Add the AssemblyAI.SemanticKernel NuGet package to your project.

dotnet add package AssemblyAI.SemanticKernel

Next, register the TranscriptPlugin into your kernel:

using AssemblyAI.SemanticKernel;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Orchestration;

// Build your kernel
var kernel = new KernelBuilder().Build();

// Get AssemblyAI API key from env variables, or much better, from .NET configuration
string apiKey = Environment.GetEnvironmentVariable("ASSEMBLYAI_API_KEY")
  ?? throw new Exception("ASSEMBLYAI_API_KEY env variable not configured.");

var transcriptPlugin = kernel.ImportSkill(
    new TranscriptPlugin(apiKey: apiKey),
    TranscriptPlugin.PluginName
);

Usage

Get the Transcribe function from the transcript plugin and invoke it with the context variables.

var function = kernel.Skills
    .GetFunction(TranscriptPlugin.PluginName, TranscriptPlugin.TranscribeFunctionName);
var context = kernel.CreateNewContext();
context.Variables["audioUrl"] = "https://storage.googleapis.com/aai-docs-samples/espn.m4a";
await function.InvokeAsync(context);
Console.WriteLine(context.Result);

The context.Result property contains the transcript text if successful.

You can also upload local audio and video file. To do this:

• Set the TranscriptPlugin.AllowFileSystemAccess property to true
• Configure the path of the file to upload as the filePath parameter

var transcriptPlugin = kernel.ImportSkill(
    new TranscriptPlugin(apiKey: apiKey)
    {
        AllowFileSystemAccess = true
    },
    TranscriptPlugin.PluginName
);
var function = kernel.Skills
    .GetFunction(TranscriptPlugin.PluginName, TranscriptPlugin.TranscribeFunctionName);
var context = kernel.CreateNewContext();
context.Variables["filePath"] = "./espn.m4a";
await function.InvokeAsync(context);
Console.WriteLine(context.Result);

If filePath and audioUrl are specified, the filePath will be used to upload the file and audioUrl will be overridden.

Lastly, you can also use the INPUT variable, so you can transcribe a file like this.

var function = kernel.Skills
    .GetFunction(TranscriptPlugin.PluginName, TranscriptPlugin.TranscribeFunctionName);
var context = await function.InvokeAsync("./espn.m4a");

Or from within a semantic function like this.

var prompt = """
             Here is a transcript:
             {{TranscriptPlugin.Transcribe "https://storage.googleapis.com/aai-docs-samples/espn.m4a"}}
             ---
             Summarize the transcript.
             """;
var context = kernel.CreateNewContext();
var function = kernel.CreateSemanticFunction(prompt);
await function.InvokeAsync(context);
Console.WriteLine(context.Result);

If the INPUT variable is a URL, it'll be used as the audioUrl, otherwise, it'll be used as the filePath. If either audioUrl or filePath are configured, INPUT is ignored.

All the code above explicitly invokes the transcript plugin, but it can also be invoked as part of a plan. Check out the Sample project which uses a plan to transcribe an audio file in addition to explicit invocation.

Notes

• The AssemblyAI integration only supports Semantic Kernel with .NET at this moment. If there's demand, we will extend support to other platforms, so let us know!
• Semantic Kernel itself is still in pre-release, and changes frequently, so we'll keep our integration in pre-release until SK is GA'd.
• Feel free to file an issue in case of bugs or feature requests.