In part 1 of this blogpost, we gave you an overview of what a conversational tour guide iOS app might look like built on Cloud Machine Learning APIs and API.AI. We also demonstrated how to create API.AI intents and contexts. In part 2, we’ll discuss an advanced API.AI topic — webhook with Cloud Functions. We’ll also show you how to use Cloud Machine Learning APIs (Vision, Speech and Translation) and how to support a second language.
Webhooks via Cloud Functions
In API.AI, Webhook integrations allow you to pass information from a matched intent into a web service and get a result from it. Read on to learn how to request parade info from Cloud Functions.- Go to console.cloud.google.com. Log in with your own account and create a new project.
- Once you’ve created a new project, navigate to that project.
- Enable the Cloud Functions API.
- Create a function. For the purposes of this guide, we’ll call the function “parades”. Select the “HTTP” trigger option, then select “inline” editor.
- Don’t forget to specify the function to execute to “parades”.
You’ll also need to create a “stage bucket”. Click on “browse” — you’ll see the browser, but no buckets will exist yet. - Click on the “+” button to create the bucket.
- Specify a unique name for the bucket (you can use your project name, for instance), select “regional” storage and keep the default region (us-central1).
- Click back on the “select” button in the previous window.
- Click the “create” button to create the function.
The function will be created and deployed: - Click the “parades” function line. In the “source” tab, you’ll see the sources.
Here’s our package.json file. This is dependent on the actions-on-google NPM module to ease the integration with API.AI and the Actions on Google platform that allows you to extend the Google Assistant with your own extensions (usable from Google Home):
In the index.js file, here’s our code:
In the code snippets above:
{
"name": "parades",
"version": "0.0.1",
"main": "index.js",
"dependencies": {
"actions-on-google": "^1.1.1"
}
}In the index.js file, here’s our code:
const ApiAiApp = require('actions-on-google').ApiAiApp;
function parade(app) {
app.ask(`Chinese New Year Parade in Chinatown from 6pm to 9pm.`);
}
exports.parades = function(request, response) {
var app = new ApiAiApp({request: request, response: response});
var actionMap = new Map();
actionMap.set("inquiry.parades", parade);
app.handleRequest(actionMap);
};In the code snippets above:
- We require the actions-on-google NPM module.
- We use the
ask()method to let the assistant send a result back to the user. - We export a function where we’re using the actions-on-google module’s
ApiAiAppclass to handle the incoming request. - We create a map that maps “intents” from API.AI to a JavaScript function.
- Then, we call the
handleRequest()to handle the request. - Once done, don’t forget to click the “create” function button. It will deploy the function in the cloud.
tell() and ask() APIs. tell() will end the conversation and close the mic, while ask() will not. This difference doesn’t matter for API.AI projects like the one we demonstrate here in part 1 and part 2 of this blogpost. When we integrate Actions on Google in part 3, we’ll explain this difference in more detail.
As shown below, the “testing” tab invokes your function, the “general” tab shows statistics and the “trigger” tab reveals the HTTP URL created for your function:
Your final step is to go to the API.AI console, then click the Fulfillment tab. Enable webhook and paste the URL above into the URL field.
With API.AI, we’ve built a chatbot that can converse with a human by text. Next, let’s give the bot “ears” to listen with Cloud Speech API, “eyes” to see with Cloud Vision API, a “mouth” to talk with the iOS text-to-speech SDK and “brains” for translating languages with Cloud Translation API.
Using Cloud Speech API
Cloud Speech API includes an iOS sample app. It’s quite straightforward to integrate the gRPC non-streaming sample app into our chatbot app. You’ll need to acquire an API key from Google Cloud Console and replace this line in
Follow this example to use Cloud Vision API on iOS. You’ll need to replace the label and face detection with landmark detection as shown below.
SpeechRecognitionService.m with your API key.#define API_KEY @"YOUR_API_KEY"Landmark detection
NSDictionary *paramsDictionary =
@{@"requests":@[
@{@"image":
@{@"content":binaryImageData},
@"features":@[
@{@"type":@"LANDMARK_DETECTION", @"maxResults":@1}]}]};Follow this example to use Cloud Vision API on iOS. You’ll need to replace the label and face detection with landmark detection as shown below.
You can use the same API key you used for Cloud Speech API.
Text to speech
iOS 7+ has a built-in text-to-speech SDK, AVSpeechSynthesizer. The code below is all you need to convert text to speech.#import
AVSpeechUtterance *utterance = [[AVSpeechUtterance alloc] initWithString:message];
AVSpeechSynthesizer *synthesizer = [[AVSpeechSynthesizer alloc] init];
[synthesizer speakUtterance:utterance]; Supporting multiple languages
Supporting additional languages in Cloud Speech API is a one-line change on the iOS client side. (Currently, there's no support for mixed languages.) For Chinese, replace this line in SpeechRecognitionService.m:recognitionConfig.languageCode = @"en-US";recognitionConfig.languageCode = @"zh-Hans";To support additional text-to-speech languages, add this line to the code:
#import
AVSpeechUtterance *utterance = [[AVSpeechUtterance alloc] initWithString:message];
utterance.voice = [AVSpeechSynthesisVoice voiceWithLanguage:@"zh-Hans"];
AVSpeechSynthesizer *synthesizer = [[AVSpeechSynthesizer alloc] init];
[synthesizer speakUtterance:utterance]; 
Next time
We hope this example has demonstrated how simple it is to build an app powered by machine learning. For more getting-started info, you might also want to try:
In part 3, we’ll cover how to build this app on Google Assistant with Actions on Google integration.
- Cloud Speech API Quickstart
- Cloud Vision API Quickstart
- Cloud Translation API Quickstart
- API.AI QuickstartY
In part 3, we’ll cover how to build this app on Google Assistant with Actions on Google integration.
