Bots sample with speech recognition and synthesis

Some posts ago I wrote about the Microsoft Bot Framework, giving some useful links. Among others, there were the following, related to bots examples with Cortana, speech recognition and synthesis … there are not so many references online:

In this post I will give some more details related the TriviaBotSpeechSample in order to make it run and see most of the features related to speech recognition and synthesis. It is a very complex examples that make use of many features, like embed the bot in a custom UWP app, showing how to add the speech support in an app using the DirectLine channel. This channels allows to interact with the bot at a low level with specific APIs: the library supports both UWP and C# applications in XAMARIN, to allow developers to include speech enabled conversations with bots across different platforms (native iOS, Android, Windows).

A good tutorial of that sample can be found in the bot framework portal (Speech Support in Bot Framework – Webchat to Directline, to Cortana) and the more relevant instructions to make it be deployed and see it work can be found in the instructions available in the GitHub repository of that solution. As said there, this sample contains two projects, a trivia bot (built on top of the Microsoft Bot Framework C# Builder SDK) and a UWP app (that talks to the bot using the Microsoft.Bot.Client NuGet package).

As this is a combined client/sample demo, there is a bit of setup required and in the following I will improve the available instructions available there, with some screenshots I did during the test I did together with a friend, … that possibly can help you! 😉

At the end , you can talk to the sample bot in multiple ways, all of which support speech input and output … I succeeded to make it run with all of them:

Using the TriviaApp (UWP app) included in this sample and simply hitting F5 in Visual Studio to run it.
Using the bot framework emulator https://docs.microsoft.com/en-us/bot-framework/debug-bots-emulator.
Host an instance of the Bot Framework WebChat client: https://aka.ms/BfWebChat
Enabled the bot as a Cortana Skill. Simply enable the Cortana channel and provide an invocation phrase. Then make sure you are logged in to Cortana using the same Microsoft account, and say “Ask to start a game of trivia“. Cortana should trigger that bot!

========================================================================

First of all, be sure to update all libraries used in the solution, going in Visual Studio Tools -> NuGet Packet Manager -> Manage Nuget Packeges for Solution: otherwise, especially if you will use the latest emulator version, you will have the error shown in the following screenshot-.
Note that only one library (System.IdentityModel.Tokens.Jwt) cannot not be updated to the latest version because of some dependences that require a lower version of that package.

Register a new bot with the Bot Framework at http://dev.botframework.com/ and add the AppId and AppPassword to TriviaBot\Web.config.

In order to link the bot of the sample, it is convenient to choose the seccond options, otherwise it will be created and registered in Azure a new “basic” bot (and then you have to download the solution and substitute it with the one of the sample … so more wotk!).

If you are trying to create a ID/password with your MSDN subscription done by your company, you will possibly receive the following error due to insufficient privilege: because I needed that codes only for a trial and not to publish a company app, I solved the problem generating them with my personal Microsoft account.

You need to set the bot handle in order it is unique, because the URL will be realoted to that name:

Enable the Direct Line channel in the bot settings page, Add a new site and paste a Direct Line secret in TriviaApp\BotConnection.cs file.

Create a LUIS app on http://luis.ai/ and import TriviaBotLU.json (available in the sample code) as a new app. Train and publish the model, then add the app id and subscription key in TriviaBot\TriviaDialog.cs (there is an error pointing to the location). The LUIS app id and subscription key can be extracted from the Endpoint Url provided on the “Publish App” page at http://luis.ai/ The link format is: https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/yourappid?subscription-key=yoursubscriptionkey&verbose=true&q=.

Reference to Bing recognition API, as requested by the code and related generated error.

Publish the Bot as an Azure App Service, and add the public endpoint (yourhosturl/api/messages) to the Bot Framework portal settings page. Make sure to use https instead of http in the url.

[Optional] To improve speech recognition for your bot: on http://dev.botframework.com/ go to the bot’s settings. In the “Speech recognition priming with LUIS” section you should see a list of LUIS apps associated with the account you are logged in with. Check the new LUIS app you created for this bot and hit save. This information is used to improve speech recognition when you speak to this bot and uses the Cognitive Speech apis for speech recognition. Speech recognition priming improves the recognition accuracy for the utterances and entities defined in your LUIS app for this bot.