Rainbird and Microsoft Cognitive Services

  • BLOG POST
  • 14/06/2017
During a three day hack-a-thon in April 2017, we experimented with integrating a range of Microsoft Cognitive Services with the Rainbird Platform. Using a demo Knowledge Map built in Rainbird, we used Microsoft Cognitive Services to produce bots in Skype and Slack.

Developers: Chris Holden and Will Layton

Infrastructure: Ramil Brion

Author: Nathan Roberts

Date: 26/04/2017

Blog Introduction
During a three day hack-a-thon in April 2017 we experimented with integrating a range of Microsoft Cognitive Services with the Rainbird Platform. Using a demo Knowledge Map built in Rainbird referred to as the Hamilton demo we used Microsoft Cognitive Services to produce bots in Skype and Slack. The objective of the Hamilton demo is to enable Bank Account recommendation through rules generated by banking experts. More details on the Rainbird platform and the Hamilton demo are available through the links under ‘Resources’ further down.

In addition we deployed a Rainbird environment on the Azure cloud hosted Platform. A Rainbird environment consists of database instances, an inferencing engine and the expert modelling tool. Each required element is available as a docker image and therefore simple to deploy.

Tools used during hack-a-thon
Microsoft Bot Framework
Microsoft Cognitive Services:
LUIS
QnA Maker
Rainbird Platform

Challenge
Rainbird’s power is best demonstrated by taking a complex decision making routine generally performed by teams of people and automating that decision making process in a tool for others to use. Very often a customer’s requirement encompasses this scenario along with a need to answer basic questions. Microsoft’s QnA Maker extracts frequently asked questions and answers, making them accessible through it’s API. Additionally customers frequently ask about Rainbird bot integration therefore we have used Microsoft’s Bot Framework to provide a front end in both Slack and Skype. Finally we integrated LUIS, Microsoft’s Natural Language Processing (NLP) tool, to identify the user’s free text input intent from which the application routes requests through QnA Maker or Rainbird.

Solution

Overview

In summary the user’s initial request is passed to LUIS to identify whether the request for information should be directed to QnA Maker or Rainbird. When QnA Maker is identified as the route the API is contacted and the answer communicated back to the user. When Rainbird is identified as the route an iterative Q & A consultation is conducted to solve the user’s original query. The consultation is directed to aid the interaction using the attributes of the median (Skype, Slack, etc.) as illustrated below:

The solution’s code is publicly accessible here.

Testing the solution

QnA Maker

In QnA Maker we defined a handful of questions and associated answers as shown below:

Rainbird

In Rainbird we defined a single complex decision task to recommend a suitable bank account. You can ask “Please recommend me a bank account” or a variance of this request as LUIS will determine the intent. Our solution will then interact with Rainbird asking a series of questions before making a recommendation. The final recommendation is supplemented with a graphical representation of the path taken, the Rainbird Evidence Tree.

Try it out

The Microsoft bot framework by default generates Web Chat and Skype implementations,
click here to try out the final solution in the Web Chat.

Step by Step Guide

Here’s a quick explanation to assist reproducing the solution. To use the Microsoft services you will need to sign up to each of them:

LUIS
Azure
Bot Framework
QnA Maker

The solution requires a number of environment variables associated with the above services and Rainbird. When deploying your bot in the Azure Infrastructure use the Azure portal to enter the required environment variables.

The application’s code is primarily in app.js. We’ve additionally developed Metaintent.js, a wrapper to handle the user’s input so we can determine the best tool to solve their question.

Firstly in app.js we create a server using restify.js as a means of easily capturing the Rest API requests in our application and configure this to listen on a desired port:

Our setup continues with the Microsoft Bot Framework dependency, botbuilder. From which we can retrieve a universal bot instance used to navigate our conversation through dialogs.

Our attention then turns to configuring our connection to Rainbird. Below we have informed the rainbird api (URL), of our API key and the ID of the Knowledge Map, thus identifying ourselves and the knowledge map we wish to query.

Our Rainbird knowledge map is able to run a single query which is ‘recommend a bank account’. Therefore we have preconfigured the query JSON to this effect using environment variables.

Our application’s initial setup is completed through the use of Microsoft’s universal bot connector which instructs our application to listen for messages.

Click here to learn more about the default message handler.

Besides a number of helper functions designed to handle the Rainbird interactions there are seven bot.dialog() handlers. The default message handler uses dialog to control the conversation flow. See here for more details.

The root dialog handler below decides on one of three courses of action based on the result of calling metaIntent.process(). If you recall we mentioned that the metaIntent.js was used to determine the path to take. Here we pass the user’s input through to our helper in metaIntent.js where the content is used to identify if we are asking a simple Question to be handled by Microsoft’s QnA Maker or if we are making a Rainbird request to recommend a bank account based on a series of questions and answers. As a fail safe we check for neither action and pass back a ‘Sorry didn’t understand…’ message.

Before we run the code to determine the path to take we execute the line session.sendTyping(). This line tells the bot to inform our user that we are typing.

To help capture variations to text indicating the need to run Rainbird’s ‘Recommend me a bank account’ request, we filter the user’s input through a trained LUIS implementation. To understand this lets turn our attentions to the metaintent.js content. Beginning with the processTextLuis() function we can see that LUIS is informed of the user’s input. If the response from LUIS exceeds a high enough score we return the first element in the array received from LUIS (remember we only have one intent).

The function exposed to our main app.js code attempts the LUIS intent match using the above function before evaluating the correct path to take. Assuming no error is returned from processTextLuis() a result object informs our calling code to take the Rainbird route, otherwise we attempt the QnA maker route.

Our processTextQnA() function contacts Microsoft’s QnA Maker with a formed URL, JSON body containing the user’s text, and our QnA Maker credentials (defined in environment variables). On success of a QnA Maker match the associated answer forms our result object and is passed back to our processText() function and subsequently our calling function in app.js. Again note we check for a high certainty before confirming a match ‘response.body.score > 80’.

Returning to our calling code in app.js we can see where we begin the more complex task of processing the Rainbird query by redirecting to our ‘/prestart’ dialog.

The ‘/prestart’ dialog checks for the existence of a running Rainbird conversation and redirects to our main rainbird loop dialog when one exists (‘/rbloop’). Otherwise we start a conversation by redirecting to the ‘/start’ dialog.

The ‘/start’ dialog as the name suggests starts our Rainbird conversation. This is achieved through two steps. The first calls the Rainbird API ‘start’ endpoint where we use our Rainbird credentials (API key and Knowledge Map ID) to authenticate. Next we call the ‘query’ endpoint to instruct Rainbird to start the ‘recommend me a bank account’ predefined query.

Note we’ve used a publicly available dependency to help abstract away much of the Rainbird Rest API syntax. See github for details.

Once complete our ‘/start’ dialog redirects to the ‘/rbloop’ dialog. Essentially all Rainbird paths lead to the ‘/rbloop’ dialog where we iterate through questions until Rainbird has sufficient knowledge to return result(s) relating to the original query. That is unless Rainbird exhausts all possibilities and concludes it is unable to answer the user’s query.

Finally let’s examine the ‘/rbloop’ dialog that controls much of the Rainbird discussion flow.

Before we look into the two functions in our array, let’s take a look at the last function, cancelAction(). Once a user has entered into a Rainbird conversation their input will always get redirected to the ‘/rbloop’ dialog. With this function in place if the user wishes to exit this line of conversation they can do so by typing ‘restart’. The function removes the conversation data from the session and returns the message ‘No problem, how else can I help you?’. Now the user is free to enter into a new line of conversation.

Now let’s look at the functions in the array. In our first function we check the conversation data to see if Rainbird has a question for the user. If so then we call sendRBQuestion().

Our sendRBQuestion() function examines the question content received from Rainbird to help form a suitably illustrated question i.e. question with multiple choice options. We’ve used builder.Prompts() to present the question to our bot user.

Our second function handles the bot user’s response to a question. Taking this response we contact Rainbird again this time via the yolandaResponse() function which calls the ‘response’ Rainbird API endpoint. Similarly to when we called query we consider the result of calling the endpoint to determine if we need to ask another question by redirecting to ‘/rbloop’ dialog or if we have a result to our original request.

When the Rainbird response is not a question we process the response as an answer to our query. First we call sendRBResult() which constructs our message to inform the user of the query results. We then clear out our conversation to allow future Rainbird queries to run without knowledge of the answers given in this conversation and finally we complete our clean up by instructing our bot to endDialog().

In the above sendRBResults() we construct a link to Rainbird’s Evidence Tree, a visual representation of the path Rainbird took to derive the associated answer. An optional display which in some industries is essential for compliance.

To complete our analysis of the solution, let’s look at the code used by Skype and Web Chat to present a welcome message. Note this feature is not available in Slack.

As you can see it’s possible to handle bot specific events. Here we have demonstrated handling both Web Chat and Skype specific events to present the user with a ‘welcome’ message when they first enter the bot.

Our final step of the hack was to configure and deploy our bot implementation in Azure. We followed this blog to guide us through the deploy process.

Summary

This demo produced in the three day hack represents a starting point that other implementations can quite easily build on inline with specific requirements. You could go on to add additional Rainbird queries identified through further training in LUIS. Expand the range of QnA Maker questions and even configure multiple Rainbird knowledge maps. A solution that is scalable yet flexible.

Technical limitations such as the handling of the ‘restart’ command could be improved on perhaps by supporting multiple conversations simultaneously.

The bot framework through Azure makes it easy to roll out to other bots (referred to as channels in Azure), by default Skype and Web Chat are selected, but adding others is as easy as signing up, selecting the channel and configuring with your credentials. This is all achieved in the bot framework portal. The array of available channels is extensive; Bing, Cortana, Facebook Messenger, Kik, and Slack, as well as several others.

Our only stumbing block came from the bots inability to handle plural answers when presenting a multiple choice question. When the bot presented options the functionality followed radio button behaviour, whereas our use case would have favoured check box behaviour. Having spoken with Microsoft they assure us this is being worked on and we look forward to seeing this feature in a future release.

If you would like details on how you could benefit from Rainbird please contact us.

Resources

Contact us at Rainbird.ai.

See how the Rainbird Hamilton Demo was built.

Microsoft QnA Maker

Microsoft Bot Framework

Microsoft Bot Framework (repository)

Hack’s code repo

https://rainbird.ai/user-guide/contents/

Restify

Link used to guide us through deploying the bot in Azure.

Additionally we found this resource useful when bringing together the tools.