mental algorithm is used for gesture recognition. KEYWORDS: modeless pen-based user interface, continu- ous feedback, incremental gesture recognition.
Using Incremental Gesture Recognition to Provide Immediate Feedback while Drawing Pen Gestures Peter Tandler, Thorsten Prante GMD – German National Research Center for Information Technology IPSI – Integrated Publication and Information Systems Institute AMBIENTE – Workspaces of the Future Dolivostr. 15, D-64293 Darmstadt, Germany {Peter.Tandler, Thorsten.Prante}@darmstadt.gmd.de http://www.darmstadt.gmd.de/ambiente/ ABSTRACT
RELATED WORK
As recognition of pen gestures tends to be ambiguous and error-prone, it is essential that users get feedback about recognized shapes as early as possible. This is especially important in modeless user interfaces with no separate modes for writing and gesturing. In this paper, we propose to give feedback continuously while users draw, as this allows immediate correction of minor mistakes. To be able to provide the speed necessary for fluent writing, an incremental algorithm is used for gesture recognition.
Many existing systems simply ignore the problem of dealing with recognition errors. Even if ambiguity is handled, this happens after pen strokes have been entered completely, not enabling the user to correct the input while drawing [1]. Zhao [6] has published an incremental recognition algorithm for syntax-directed diagram editors. But this algorithm incrementally combines several strokes to multi-stroke gestures. Fluid Sketches [5] provide continuous feedback about simple geometric shapes the user draws by morphing the drawn stroke to the recognized shape. While this is an excellent way of giving feedback, their algorithm uses differential equations that have to be solved by a concurrent thread. As they state, “the fluid sketching approach is feasible for simple geometric shapes”. The algorithm proposed in this paper can be used for arbitrary shapes.
KEYWORDS: modeless pen-based user interface, continuous feedback, incremental gesture recognition INTRODUCTION
As devices without mouse or keyboard are becoming widely used, new interaction techniques adapted to their needs evolve. Pen gestures are often used to benefit from the natural interaction with pens. To be able to invoke commands using gestures, the raw input events have to be assembled and interpreted. The interpretation of the user input is difficult, as it can be ambiguous leading to recognition errors if the wrong interpretation is chosen. This reduces user acceptance and effectiveness. In modeless user interfaces, correct interpretation is especially important, as the shape is used to distinguish between commands and scribbles. This paper proposes to give immediate feedback to the user while drawing. As writing speed is crucial for usability of pen based systems, an incremental recognition algorithm is used. After a discussion of related work, the incremental recognition algorithm is explained. Concluding, first experiences and plans for future work are described.
INCREMENTAL RECOGNITION
To illustrate the immediate feedback, we use the example of a user drawing a box shape (fig. 1). For simplicity, the gesture recognizer used in this example only supports three different shapes: line shape, “L” shape, and box shape. The user starts drawing a line (a), which is recognized as a known shape. To give feedback about the recognized shape, the stroke turns red. When the user reaches the first angle (b), the color changes to the current scribble color again. As an “L” shape has a minimum length of both line segments (c), the “L” shape is recognized until the user comes to the next turn (d). Finally, when the end point comes close to the start point again, a box shape is identified (e).
(a)
LEAVE BLANK THE LAST 2.5cm OF THE LEFT COLUMN ON THE FIRST PAGE FOR US TO PUT IN THE COPYRIGHT NOTICE!
(b)
(c)
(d)
(e)
Figure 1: Intermediate steps while a user is drawing a box shape. If a shape is recognized the color changes to red (a, c, e). Design of the Recognizer
Figure 2 shows the objects involved in the recognition process. The recognizer is responsible for transforming a se-
quence of pen (or mouse) positions into a set of matching shapes. It uses a set of features that are computed from the sequence of points to determine a set of matching shapes, similar to Rubine’s algorithm [4]. While it is active, the stroke tracker, used for drawing strokes with a pen, receives events directly from the event dispatcher. Recognition Process
When the tracker is started (upon a pen down event), it creates a new instance of the recognizer. While the stroke tracker is active, besides drawing and recording the stroke, it sends all points immediately to its recognizer. When the recognizer receives a new position, it first updates all of its features. The features used in this example (see fig. 3) are key points (i.e. the corners of a stroke), the length of the stroke, and the bounding box (i.e. the minimal box containing the stroke). Based on the values of the features, the recognizer checks all possibly matching shapes, whether they match or not. For example, a line shape needs to have two key points, and the distance from the first to the last point should be similar to the length of the line feature. aStrokeTracker
1. send mouse events
keypoints lengthOfLine boundingBox
keypoints
aLineShape aLShape aBoxShape aScribbleShape (a)
bounding box
aLShape aBoxShape aScribbleShape (b)
aLShape aBoxShape aScribbleShape
aBoxShape aScribbleShape
(c)
(d)
aBoxShape aScribbleShape
(e)
Figure 3: The features are updated incrementally while the stroke is drawn, if shapes cannot match anymore, they are removed from the recognizer.
With the updated values, the recognizer asks all shapes for their current similarity to the stroke. If a shape returns zero, e.g. because the stroke has too many key points, the recognizer removes it from the list of possibly matching shapes (fig. 3 a–b, b–c, and c–d). The scribble shape is the default shape and will never be removed from that list. EXPERIENCES AND FUTURE WORK
aRecognizer
2. update features
easy as in the previous examples. Depending on whether the last point is a corner or not, the last key point is replaced (see fig. 3 b–c).
3. update matching shapes
aLineShape aLShape aBoxShape
Figure 2: The recognizer receives mouse events from the tracker and incrementally updates the features and list of matching shapes.
After all shapes are updated, the tracker checks whether the currently most likely shape has changed. In this case, it asks the new shape for its associated stroke color and redraws the stoke. Finally, when the stroke is finished (pen up event), the tracker creates a stroke event associated with the final shape and passes it to the event dispatcher for further processing. Incremental Update
While our first implementation recomputed all features every time a new point was added to the stroke, we figured out that this was too slow to be acceptable. Therefore we changed the recognizer to update the features incrementally: The recognizer now sends every new point to every feature, which will merge the new value with the previous state. Each feature can decide whether to update or recompute. • The length of the stroke is computed by summing the distances between all points. It is much faster only to add the new distance. • The bounding box needs to check whether the new point is outside the previous box and enlarge it if necessary. • To compute the key points, it is necessary to look for turns in the stroke. As the first and last point of the stroke are always key points, the incremental update is not as
The algorithm proposed in this paper is used for the scribble and gesture interface of BEACH [3], the software infrastructure of i-LAND [2]. Using BEACH for our work and for presentations, we found that the writing speed is crucial for fluid interaction. This lead us to the incremental algorithm. Immediate feedback is very helpful as gesture recognition is inherently error-prone. Thus a very fast recognition algorithm helps improving the usability of the system. Currently, we are implementing audio feedback for recognized shapes. In the future, we will extend our implementation to support trainable shapes, as described in [4]. To gain better recognition rates we will investigate how to support personalized gestures in a multi-user environment like i– LAND. ACKNOWLEDGMENTS We would like to thank all our colleagues for helpful discussions, and Oliver Seiler for implementing the recognition algorithm. REFERENCES 1. Mankoff, J. et al. Providing integrated toolkit-level support for ambiguity in recognition-based interfaces. Proc. of CHI 2000, pp. 368–375. 2. Streitz, N. et al. i-LAND: An interactive landscape for creativity and innovation. Proc. of CHI 1999, pp. 120-127. 3. Tandler, P. Software Infrastructure for a Ubiquitous-Computing Environment Supporting Collaboration with Multiple Singleand Multi-User Devices. Submitted to UbiComp 2001. 4. Rubine, D. Specifying Gestures by Example. SIGGRAPH 25(4), pp. 329–337, 1991. 5. Arvo, J., Novins, K. Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes. Proc. of UIST 2000, pp. 73–80. 6. Zhao, R. Incremental recognition in gesture-based and syntaxdirected diagram editors. Proc. of INTERCHI 1993, pp. 95–100.
