[Back to Skill Manifest Guide]
To understand natural language, Genie generates hundreds of thousands of possible commands based on a general grammar of English, a few annotations, and domain-specific templates for each Thingpedia device.
Developers can provide domain-specific information for their Thingpedia devices by adding:
canonical forms
associated with each API element in the manifest.tt
fileprimitive templates
to annotate entire snippets of ThingTalk with a short phrase in the dataset.tt
file.Canonical forms are short phrases defining how to compose commands that use a specific API element in Thingpedia. They can be attached to a class, a function, or a parameter using the #_[canonical]
natural language annotation.
Please refer to the genie-annotation guide for the full details on the canonical forms.
Now, let's first see a couple of examples and we will explain the concept as we walk through it.
Suppose we want to create a Yelp skill that allows users to query restaurant information and it should return a list of restaurants with the following information:
id
: Name of the restaurantlink
: Restaurant's website URLcuisines
: Restaurant's category or type of food it servesprice
: Restaurants' price levelsrating
: Restaurant's review ratinggeo
: Restaurant's locationphone
: Restaurant's contact numberopening_hours
: Restaurant's opening hoursWe can create a manifest.tt
file for Yelp like this:
class @com.yelp
#_[canonical="yelp"]
{
entity restaurant #_[description="Restaurants on Yelp"];
entity restaurant_cuisine #_[description="Cuisines in Yelp"];
list query restaurant(out id: Entity(com.yelp:restaurant)
#_[canonical={
default="base",
base=["name"]
}],
out link: Entity(tt:url)
#_[canonical="link"],
out cuisines: Array(Entity(com.yelp:restaurant_cuisine))
#_[canonical={
default="adjective",
base=["cuisines", "category"],
property=["# food"],
adjective=["#"],
verb=["serves # cuisine", "serves # food"],
preposition=["in the # category"],
base_projection=["food", "cuisine"],
verb_projection=["serve", "offer", "have"],
}],
out price : Enum(cheap, moderate, expensive, luxury)
#_[canonical={
default="adjective",
base=["price range", "price"],
property=["# price"],
preposition=["in the # price range"],
adjective=["#"]
}],
out rating: Number
#[min_number=1]
#[max_number=5]
#_[canonical={
default="passive_verb",
base=["rating"],
passive_verb=["rated ${value} ${value:plural: one{star} other{stars}}"],
adjective=["${value} ${value:plural: one{star} other{stars}}"],
property=["rating", "${value} star rating"],
adjective_argmax=["best", "top rated"],
adjective_argmin=["worst"],
passive_verb_projection=["rated"]
}]
#_[counted_object="stars"],
out geo: Location
#_[canonical={
default="preposition",
base=["address", "location"],
preposition=["at #", "near #", "in #", "around #"]
}],
out phone: Entity(tt:phone_number)
#[filterable=false]
#_[canonical={
default="property",
base=["phone number", "telephone"],
property=["phone number #"],
verb=["can be reached at #"]
}],
out opening_hours: RecurrentTimeSpecification
#_[canonical={
default="property",
base=["opening hours", "business hours", "hours"],
verb=["opens #"]
}]
)
#_[canonical=["restaurant", "eatery", "place to eat", "food place"]];
}
The canonical form of each class defines how that class is referred to in natural language. Its value needs to be a string.
Example:
Line
2
shows that Yelp skill can be referred to asyelp
. Genie will use this term to generate commands and users can simply use the termyelp
to query restaurants.
Changing Line
2
to#_[canonical="yippee"]
will change the restaurant query invocation toyippee
.
The canonical form of each function defines how that function is referred to in natural language. Its value can be a string, or an array of strings.
Example:
Line
75
shows a list of phrases, andrestaurant
,eatery
,place to eat
, andfood place
are the terms that refer to a restaurant query, which will return a list of restaurants. Genie will use these terms to generate commands and users can also invoke restaurant list queries with any of these words.
The canonical form of each parameter defines how the parameter is specified to the action, or how it can be used in natural language to filter the result of the query. They are defined as object, whose keys are as follows:
base
: the base form the of the parameter, used in explicit filters and explicit projections. (The definition of database operations can be found here.Example:
In line
45
, the restaurant's rating hasrating
in the base form. So when explicitly filtering restaurants that are rated 5 stars, the command can be in the form ofShow me all restaurants with rating equal to 5 stars
. If explicitly asking for a projection on the restaurant's rating, the command can be something likeWhat is the rating of Ruth's Chris?
.
#
character replaces the value.Example:
In line
20
, restaurant'scuisines
parameter has the term# food
as the property and uses# food
to describe what the restaurant has. Genie can use this to generate commands likeCan I find a place that serves Chinese food?
to select a specific type of cuisines, where#
is replaced byChinese
in this case.
_argmin
or _argmax
, which is used to form a ranking question.Example:
In line
46
and47
,best
andworst
terms are used to describe how a single adjective phrase can be used to get the highest or least rated restaurants, respectively. Genie can simply generate commands likeShow me the best restaurants nearby
for querying top rated restaurants.
default
: the grammatically preferred form to use, which is then given slight priorityExample:
In line
18
,adjective
form (i.e.Italian cuisine
,French cuisine
) will be used at a higher priority when commands are related to restaurant'scuisines
.
All canonical forms should be lowercase and should not use any punctuation. No placeholder can be use other than the
#
for a filter canonical form.
Below are some representative commands that Genie can understand and generate based on the canonical forms defined in the example:
Let's create a Spotify skill that helps users to create their favorite lists. It basically only requires an input parameter:
playlist
: Name of the favorite listWe can create a manifest.tt
file in this way:
class @com.spotify
#_[canonical="Spotify"]
{
action create_playlist(in req name : String
#[string_values="com.spotify:playlist"]
#[raw_mode=true]
#_[prompt=["what do you want to name your playlist"]]
#_[canonical={
default="base",
base=["name"],
preposition=["named #", "called #", "titled #"],
property=["name #", "title #"]
}])
#_[canonical=["create a new playlist", "create playlist"]]
;
}
Example:
Line
2
shows that Spotify skill can be referred to asSpotify
. Genie will use this term to generate commands and users can simply use the termspotify
to invoke spotify functions.
Example:
Line
14
shows thatcreate a new playlist
andcreate playlist
are the terms that refer to thecreate_playlist
action. Genie will use these terms to generate commands.
Example:
In line
10
, thename
is declared as the base form, So, explicitly, the command can be in the form ofCreate a new playlist with name My Favorite
.
Below are some representative commands that Genie can generate:
On occasion, especially with actions, the general grammar templates are not sufficient to capture the variety of real-world commands. In that case, one can specify example commands in the form of primitive templates
. Primitive templates are composable snippets of natural language and ThingTalk program that allow Genie to construct complex commands (with multiple joins and filters).
Primitive templates are specified in a dataset.tt
file. Here is a subset of the dataset
used by Spotify:
dataset @com.spotify language "en" {
action (p_song :Entity(com.spotify:song)) := @com.spotify.play_song(song=p_song)
#_[utterances=["play ${p_song}",
"i would like to hear ${p_song}",
"i would like to listen to ${p_song}"]]
#[id=27924741]
#[name=""];
program (p_artist :Entity(com.spotify:artist)) :=
now => @com.spotify.song(), contains(artists, p_artist) => @com.spotify.play_song(song=id)
#_[utterances=["play ${p_artist:no-undefined}",
"play some ${p_artist:no-undefined}",
"i would like to listen to ${p_artist:no-undefined}"]]
#[id=27924875]
#[name=""];
}
With these primitive templates, and combined with the previous class definitions, Genie is now able to generate commands of the form:
In particular, primitive templates for actions allow it to compose an action with a complex query ("play" + "the most popular song released this month"). This composition is based on the common type Entity(com.spotify:song)
. Primitive templates also allow it to express common constructs that refer to queries implicitly, such as "Play Coldplay" meaning "Play songs by Coldplay".
When creating a new device, a dataset.tt
file containing the example commands is required.
These example commands provide the training data for your device.
There are four types of primitive templates, denoted with the keywords action
, query
, stream
or program
. A program
primitive template contains a stream, a query (optional), and an action. It cannot be composed further. Other primitive templates contain only one of those elements and are composed of other program parts to form the full ThingTalk command.
Primitive templates can also have parameters, which will be replaced with concrete values in the generated program. For example, given the following template:
action (p_song : Entity(com.spotify:song)) := @com.spotify.play_song(song=p_song);
we automatically generate the programs:
now => @com.spotify.play_song(song="..."^^com.spotify:song("Titanium"));
now => @com.spotify.play_song(song="..."^^com.spotify:song("She Wolf"));
now => @com.spotify.play_song(song="..."^^com.spotify:song("Chandelier"));
...
By convention, parameters in primitive templates begin with p_
, to distinguish them from function parameters.
The parameters of a primitive template can be used anywhere in the body, not just as input parameters. For example, they can be used as filters:
query (p_artist : Entity(com.spotify:artist)) => @com.spotify.song(), contains(artists, p_artist);
A primitive template cannot be executed by Genie right away, but it can be composed with other primitive templates and builtin functions (e.g., now
, notify
, timer
) to form a full program. For example, if we have the following two primitive templates:
query := @com.thecatapi.get();
action := @com.slack.send();
Genie can generate a list of full programs including:
now => @com.thecatapi.get() => notify;
now => @com.slack.send();
now => @com.thecatapi.get() => @com.slack.send();
attimer(time=...) => @com.thecatapi.get() => notify;
attimer(time=...) => @com.slack.send();
attimer(time=...) => @com.thecatapi.get() => @com.slack.send();
Thus, given the composable primitive templates, Genie will be able to generate a large number of full programs for training and thus achieve better accuracy.
An utterances
annotation is used to provide different ways of expressing a command. It takes a list of strings. In each utterance, concrete values for parameters are replaced by placeholders, which can be expressed by $param
or ${param}
, where param is the name of a declared parameter of the primitive template. The braces are needed if the parameter is immediately followed by a letter, a number, or an underscore.
You also need the braces if you want to pass an option. Available options are const
(with the syntax ${param:const}
) and no-undefined
. const
means that the placeholder must not be a parameter passed from a previous function; no-undefined
means that a placeholder cannot be replaced with a generic word such as “something” or “a certain value”. The syntax is as follows:
query (p_count :Number) := @com.thecatapi.get(count=p_count)
#_[utterances=["${p_count:const} cat pictures", "$p_count cats"]];
The utterances will be used to generate the synthetic sentences for the full programs composed of the primitive templates. For example, if we have the following two primitive templates with the corresponding utterances:
query := @com.thecatapi.get() #_[utterances=["a cat picture"];
action := @com.slack.send() #_[utterances=["send a message to Slack"];
Then when we compose the full program now => @com.thecatapi.get() => @com.slack.send();
, we will generate the synthetic sentences such as: “get/search/show me/find a cat picture, then send a message to slack”.
Placeholders in the primitive templates will be replaced with:
For example, given the following primitive templates:
query := @com.thecatapi.get() #_[utterances=["a cat picture"];
action (p_picture_url : Entity(tt:picture)) := @com.slack.send_picture(picture_url=p_picture_url) #_[utterances=["send ${p_picture_url} to Slack"];
The following sentences are generated:
Hint: To decide whether or not to use
const
, try replacing with the placeholder with “the” followed by a noun (e.g. “the message” or “the picture”). If the sentence does not flow grammatically, or it sounds awkward, then it is appropriate to useconst
. If you do so, you should also think of a different utterance that allows such parameter passing.
For example, this is the correct way to annotate a primitive template that selects the channel for Slack:
action (p_channel : Entity(tt:hashtag)) := @com.slack.send(channel=p_channel)
#_[utterances=["send a message to channel ${p_channel:const}", "send a message to ${p_channel}"]
Hint: To decide whether or not to use
no-undefined
, replace the placeholder with the word “something” (or “someone”, “somewhere”, etc.), and compare it against “anything” or “a certain thing”. If the use of “something” is likely to mean “anything”, then you shouldno-undefined
.
For example, this is how you annotate filters for Slack:
stream (p_sender :Entity(tt:username)) := monitor ((@com.slack.channel_history()), sender == p_sender)
#_[utterances=["when ${p_sender:no-undefined} messages me on slack"]]
The motivation is that sentences of the form “when someone messages me on Slack” should be interpreted the same as “when anyone messages me on Slack”, and not “when a certain person messages me on Slack”.
Hint:
no-undefined
does not make sense for required parameters, because the user always specify a specific value before the program is executed.
The rules through which Genie generates full programs from primitives template are defined in Genie.
By default, the utterances for a query should be noun phrases. When we compose the sentence, we will add generic verbs before the noun phrase such as get
, show
, search
. As in our example, we have the utterance “a cat picture” instead of “get a cat picture”.
Using noun phrases is necessary for parameter passing. Going back to the previous example of cats and Slack:
query := @com.thecatapi.get() #_[utterances=["a cat picture"];
action (p_picture_url : Entity(tt:picture)) := @com.slack.send_picture(picture_url=p_picture_url) #_[utterances=["send ${p_picture_url} to Slack"];
In addition to the sentence “get a cat picture, then send the picture to Slack”, which is long and verbose, we can also generate ”send a cat picture to Slack”. In this case, the placeholder is replaced with the entire noun phrase of the function that generates the result to send.
For streams, write the utterances as when phrases, such as “when it's raining”, “when I receive an email”. For actions, write the utterances as verb phrases in the imperative form, such as “send a message”.
Additionally, if you want to use a non-generic verb for your query, put a comma ,
before your utterance. The comma is a marker for a verb phrase and is automatically removed when generating sentences. For example, the following are the primitive templates for translation:
query (p_text :String) := @com.yandex.translate.translate(text=p_text)
#_[utterances=["the translation of $p_text", ", translate $p_text"]]
These templates (combined with a Slack template) result in the following sentences:
Note: The first utterance of each distinct example will be presented in Thingpedia Cheatsheet. So put the most common and natural utterance first in the list.
Note: Internally, the examples are not stored as
.tt
files, so any comment or formatting in the dataset file will be lost, and multiple examples with the same code will be collapsed.
Note: The dataset you provide as part of your device submission must use American English. Translations can be provided separately, and the
language
field of the dataset file is ignored.
To help Genie do a better job of handling commands with parameters, you must specify the example values for each of your parameters (of type String or Entity) when declaring the function. The syntax is #[string_values=<dataset-name>]
.
For example, the song
function for Spotify can be declared as follows:
action play_song(in req song : Entity(com.spotify:song)
#[string_values="tt:song_names"])
In this case, we tell the system to use the values in the dataset tt:song_names
as example values for the parameter song
. Then Genie will replace message
with the values in tt:song_names
randomly when generating the synthetic sentences.
You can choose any of the existing datasets from the Available String Datasets page, or submit your own, tailored to your device. If your device must understand values from an open ontology (that is, you expect the user to try out values not seen at training time), it is recommended to include at least 10,000 to 100,000 training examples.
Note that example values are necessary for both input and output parameters since an output parameter can also be used in the command as a filter.
Unlike most of the other programming languages, the choice of parameter names is important in ThingTalk: it affects the performance of the natural language translation because parameters of the same name share knowledge across devices in the neural network.
To achieve the best accuracy, we suggest using the same parameter names as other similar devices. Here are some naming conventions to follow:
picture_url
picture_url
url
; if it accepts any URL of videos, name it video_url
title
, the blurb or description description
, the URL link
, the update date updated
and the author name author
query
caption
and the picture picture_url
status
title
and body
set_power
, name the argument power
and make it of type Enum(on,off)
low
and the upper bound high
count
parameter of type Number
[Back to Skill Manifest Guide]