The Genie Dialog API allows you to integrate the Genie agent in new UIs, new platforms, or third party products such as Alexa.
This page describes only the abstract interface of the Genie agent. A concrete interface is provided over Web Sockets by Web Genie.
At a high-level, the Genie Dialog API consists of opening a conversation, sending some user input, and then receiving some output to present to the user. The conversation, which is identified in a platform specific manner, consists of all the dialog state known to the agent (such as the partial program being specified by the user, or which slot is being filled).
Each conversation consists of one or more rounds. Each round can be initiated with a user input, such as a ThingTalk program, or with a notification or question from Genie, for example a permission request initiated by a remote user.
In each round, the user provides some input, to which Genie responds with one or messages, followed by a single ask special message, which indicates that Genie is waiting for a response from the user. The null
ask special message indicates that Genie does not need a response from the user and terminates the round.
At any point, a round can be terminated by sending a cancel
special message (as a parsed message, or as textual input, except in certain cases).
Observe that in some cases (e.g. notifications) a round can be initiated and terminated by Genie completely with no interaction from the user.
The first form of input is raw text: you pass the unmodified textual command from the user.
In Web Genie, this is represented as a json object with type command
and a text
property:
{
"type": "command",
"text": "post \"lol\" on twitter"
}
A pre-parsed command is one where the interpretation of the command is passed as input to the agent. The format of this interpretation is described later.
In Web Genie, this is represented as a json object with type parsed
and a json
property holding the parsed commannd. Optionally a title
property can be added with the corresponding natural language command:
{
"type": "parsed",
"title": "Help",
"json": {
code: ["bookkeeping","special","special:help"],
entities: {}
}
}
By convention, in platforms that allow textual user input, a pre-parsed command is entered with the prefix \r
. For example:
\r bookkeeping special special:hello
Finally, you can provide the user input as a single ThingTalk program.
In Web Genie, this is json object with type tt
and a code
property holding the ThingTalk code:
{
"type": "tt",
"code": "now => @com.twitter.post(status=\"lol\");"
}
By convention, platforms with textual user input support raw ThingTalk with the prefix \t
. For example:
\t now => @com.twitter.post(status="lol");
(Observe that the quote marks are escaped in JSON but not in the input from the user)
As introduced in the Interaction Model, Genie responds to user with one or more output messages, followed by exactly one ask special message.
The following types of output exist.
A simple text message, to be displayed or spoken to the user. A text message has the following properties:
type : string = "text"
text : string
: the text to show to the usericon : string
: the device identifier to use as icon, next to the message; use https://thingpedia.stanford.edu/thingpedia/api/devices/icon/$icon
to download or display the actual image.A single picture, to be displayed to the user. A picture message has the following properties:
type : string = "picture"
url : string
: the URL of the pictureicon : string
: the device identifier to use as icon for the messageA link message, to be displayed to the user, with optional title and description. An RDL message has the following properties:
type : string = "rdl"
rdl : object
: the RDL object, which itself has:
rdl.webCallback : string
: the URL of the linkrdl.displayTitle : string
: the title of the link (optional)rdl.displayText : string
: the longer description of the link (optional)rdl.pictureUrl : string
: a picture (thumbnail) associated with the linkicon : string
: the device identifier to use as icon for the messageThis is a catch-all message type for future extensions of results, or "cards" that could be displayed in response to the user's query. A result message has the following properties:
type : string = "result"
result : object
: the result object, as specified in the #_[formatted]
annotation of the queryfallback : string
: a fallback text message, suitable for a voice interface or a client thaticon : string
: the device identifier to use as icon for the messageA "button" message is a message that allows the user to click on a pre-parsed command, for example as a suggestion for a ThingTalk program. A button message has the following properties:
type : string = "button"
title : string
: the text to display to the userjson : object
: the object corresponding to the parsed command triggered by this button.Note: you can convert a button message into a valid pre-parsed command message by changing only the type
property.
A "choice" message is a message that allows the user to choose one of a list of predefined choices. It is similar to a button message in how it is expected to be presented to the user, but while a button message has the same meaning at any point of the conversation, a choice is contextual. At the same time, a choice button can present choices from a set that does not map to any intent in Genie, for example contacts with similar names, or devices of the same type.
A choice message represents only one choice among many. When Genie needs the user to choose from a list, it will send many choice messages in a row, followed by the ask special message asking the user for input. Genie can also send other messages, such as a button message with back
intent or more
intent, between the choice messages and the ask special message.
A choice message has the following properties:
type : string = "choice"
idx : number
: the numerical index of this choice (typically a small integer, but the client code must not rely on it)title : string
: the main text of the choice button, e.g. the name of the device to choosetext : string
: a longer description of the choice (optional)Because they are contextual, choice messages must be hidden from the user after the user has replied to them. Most platform will also hide button messages after any user interaction, but this is not required.
A "link" message is a button that provides functionality that goes outside the conversation, such as configuring devices or opening the My Genie page.
The most common usage for link messages is to provide a link to the OAuth configuration page for a given device (hence the name). Link messages should not be confused with RDL messages: while RDL messages provide arbitrary links to web pages, link messages only provide navigation within the Genie app.
Link messages are commonly rendered as clickable buttons, while RDL messages are commonly rendered as textual links. Because the expected user response to a link messages (that is, configuring the related device) completes outside of the Genie conversation, most platforms do not hide link messages in response to user interaction.
A link message has the following properties:
type : string = "link"
title : string
: the main text of the buttonlink : string
: the relative URL identifying the target of the link.The following are valid link
URLs:
/user/register
: the user must register before completing the action (only valid in the anonymous API)/apps
: go to My Genie (a.k.a. My Skills, or a similar page containing all the user's devices)/devices/oauth2/$kind?name=$name
: perform OAuth configuration for a device of type $kind
; $name
is the name of the device and is provided to avoid a Thingpedia lookup/devices/configure/$kind?name=$name&controls=$controls
: perform form-based configuration for a device of type $kind
; $name
is name of the device and $controls
is the JSON serialization of the device factory (see the Thingpedia API for details)Apps are encouraged to ignore unexpected link URLs, or to redirect the user to the corresponding page at https://genie.stanford.edu.
The ask special messagge completes the current sequence of messages from Genie, and either completes the round or requires the user to provide more information before continuing.
An ask special message has the following properties:
type : string = "askSpecial"
ask : ?string
: what Genie is asking for.The following are valid values for the ask
property:
null
: complete the round; Genie is not asking for anything and is now in the default state. Note that this is the JSON value null
, not the string "null"
.yesno
: Genie asks a yes/no questionpicture
: Genie would like the user to choose or upload a picturephone_number
, email_address
: Genie expects the user to input a value of the given type (typically by choosing from the contact book)contact
: Genie asks for either a phone number or a email addressnumber
: the user should input a numberdate
: the user should choose a datetime
: the user should choose a time of dayraw_string
: the user should enter some free-form text (e.g. the body of an email message)choice
: the user should choose from the list of choice messages that Genie just sentcommand
: the user should type a command (this is only used when constructing programs interactively and is mostly free-form)generic
: Genie is expecting some other input, not in the list above. The round is not complete, but the client should present only a generic input interface.Note: after sending a raw_string
ask special message, Genie enters “raw” mode. In this mode, all text inputs from the user are treated as responses, including inputs that normally would control the interaction such as “yes”, “no” or “cancel”. To use those commands while in raw mode, you must use a pre-parsed command.
A Genie parsed command is a JSON object, with the following properties:
code : Array<string>
: a ThingTalk program in pre-tokenized form, or a bookkeeping intent specificationentities : object
: the concrete values that are used by the ThingTalk codeexample_id : number
: (optional) the ID of the example command in the database corresponding to this button; used for tracking which commands are more popularslots : Array<string>
: the parameters which have been replaced by SLOT
specifications in the ThingTalk codeslotTypes : object
: a map from parameter name to ThingTalk type, for all unspecified parameters which have been replaced by the SLOT
syntax