Paper Reading #21: Automatically Identifying Targets Users Interact with During Real World Tasks

Reference Information:
Title: Automatically Identifying Targets Users Interact with During Real World Tasks
Authors:Amy Hurst, Scott E. Hudson, Jennifer Mankoff
Presentation Venue: IUI’10, February 7–10, 2010, Hong Kong, China

Summary:
This paper discusses a technique that can be used to help analyze user actions in a wide variety of software that can help research human performance, software usability, and how computers are used daily. This is done by accurately finding the size and location of the targets users interact with in real world applications. There already exists APIs such as Microsoft Active Accessibility API that is widely used to find information about user interaction but many popular real world targets are not supported by this API. This technique is supposed to find the size and location of interactive targets that existing APIs cannot.

A) This is an example interface that the paper's API and Microsoft's API can evaluate. B) This is Microsoft's API evaluation. It can only detect 4 targets out of the 46. C) This is the hybrid technique used in the paper. It found 45 out of 46 targets which is way more accurate in this case.

Targets are found with the help of visual cues. The data gathered from these cues is used to estimate possible targets. By using machine learning techniques and first level recognizers choices can be made between these possible targets. These techniques create a statistical model that predicts which targets are correct based on information that describes a particular interaction.

A program called 'CRUMBS' captures information about interaction activities from windows events, keyboard, and mouse. This program uses the Microsoft API described above but hooks into environment to take 2 300x300 screenshots. One screenshot is before an action and the second screenshot is after the action. This is used to see if a target is selected.

This is the idea behind CRUMBS.

This is one of the errors that CRUMB can encounter.

This new method was able to identify 89% of the targets while the underlying accessibility API could only identify 74% of the targets.

Discussion:
Like several of the other papers I have recently read I like the idea of this one because its about making something that exists more efficient than it already is. The idea of this technique was to make it even easier to evaluate software usability, human performance, and other research with computers. This technique is helpful because it automatically collects the size and location of things that users interact with in real world applications. I think the pictures I put on this blog help show a little better what the idea behind the paper is because it was kind of hard for me to explain.