Paper Reading #11: A Conversational Interface to Web Automation

Comments:
Comment 1
Comment 2

Reference Information:
Title: A Conversational Interface to Web AutomationAccuracy
Authors:  Tessa Lauy, Julian Cerruti, Guillermo Manzato, Mateo Bengualid, Jeffrey P. Bigham, Jeffrey  Nicholsy
Presentation Venue: UIST’10, October 3–6, 2010, New York, New York, USA
 

Summary:
This paper is about a conversational interface named CoCo.CoCo is an assistant that performs task for the user on the web and reports back the results. The limitations to this interface are it can only perform tasks that can be carried out on the web and if focuses on tasks with known outcomes, not sense-making questions.

CoCo works by taking text from the user and then parsing the request. A planning component then uses the commands to figure out what actions on the web will be taken to satisfy the request. Scripts are used to automate certain tasks so parameters are needed in the request. After a plan is established the parameters are plugged in and then the script is routed to a browser automation server which runs the script. Relevant portions from the web pages are then returned to the user. It is also worth noting that CoCo doesn't exactly need a script to run, it can also get arbitrary scripts from CoScripter Reusable History for a user.

Taken from the paper

CoCo can be used from SMS, Twitter, email, Android application with speech input, Unix command-line, lightweight web UI, and a REST API. An example of how CoCo interacts on Twitter is as follows.
Bob has the ability to forward calls from his work phone to his home phone when he is out of the office. Bob has created a CoScripter script to automate the task. Bob leaves work but forgets to run the script. He gets on Twitter and sends a directed message to CoCo saying 'forward phone calls to home'. CoCo finds the needed script based on the input, runs the script, then tells Bob the script completed successfully.

A couple areas of concern with CoCo is preventing mistakes and securing private information. If CoCo is capable of repeating any action you have previously done on the web via the CoScripter Reusable History what stops it from purchasing something you don't want or deleting something important? One plan is to develop the ability to recognize web actions that have side effects. Regarding private information, currently CoCo trusts in third party authentication systems. With future implementation a second layer of authentication might be provided.

Discussion:
CoCo seems like a very good idea and something that would be awesome to use in some cases. At the beginning of the paper the author talks about science fiction writers have written about computers that do whatever we tell them to and respond to complex commands with limited interaction. This is a step in that direction just with limitations compared to the science fiction version. I'm interested to see how this application is developed and how future products advance this idea.