In de
Bourcier, Lemmen & Thomson eds., (1994) The Seventh
White House Papers: Graduate Research in the Cognitive and Computing Sciences
at Sussex. University of
Sussex, School of Cognitive and Computing Sciences, Brighton U.K. Research
Paper CSRP 350
Informal Interfaces
-
Informality in Human-Computer
Interaction
Ian Cullimore
School of
Cognitive and Computing Science
University of
Sussex
Brighton, BN1 9QH,
England
phone: +44 273
678195
email:
ianc@cogs.susx.ac.uk
ABSTRACT
This paper discusses the notion of informality in HCI, leading to the
design of informal interfaces. Such an interface exhibits tolerance in its input and variance in its output. Informal interface
representations are internally composed of informal objects that are a combination of a prototype, such as a straight line, and associated informal dimensions such as shakiness and thickness. In an informal
interface it is the gist of
human-computer interaction, instead of a higher level of formalism, which is
paramount. Internal representations of informal objects can be decomposed,
manipulated, and recomposed. An example is given of a software tool that has
been developed to investigate the design of informal interfaces.
KEYWORDS: informality, gist, HCI, tolerance, variance, constraint, sketch, frame.
INTRODUCTION
The aim of this paper is to outline a framework for informal interaction between a computer and a human. By using the term informal in user interface design, we do not simply mean the converse of formal, nor sloppiness. Rather we are referring to interfaces that are tolerant of the user’s input (the user has flexibility in choice of action) and that show varience in their output. More specifically, in an informal interaction there is no one-to-one mapping between an input event (e.g. a menu selection) and a state change in the notional machine, nor a simple mapping between the state of the notional machine and an output presentation. Consider the classic example of a pull-down menu system; here the user is constrained to a finite set of possibilities of function choices, each of which maps onto one state, and each such state is shown as a single presentation by the interface. Conversely, an informal interface may map a number of different input events onto a single state of the notional machine, with the states representing the gist of the interaction. The states of the notional machine may be presented in a variety of forms, governed by the constraints of the internal representation and the restrictions of the output device.
WHY INFORMAL INTERFACES?
Informality suggests a lack of precision, an easing of social or linguistic conventions. The benefits of informality include being able to express a vague or partially-understood idea, and being able to explore the essence of a concept without being committed to its eventual form. Sketching, for example, has long been recognised, from the days of Leonardo da Vinci [5], as a powerful aid to allowing the mind to run freely and creatively. The sketch serves as a framework on which the mind can build. An informal interface is an analogy of the sketch in human-computer interaction, relaxing the conventional input/output constraints of current user interfaces, in order to offer the user a more evocative and richer environment in which to react and be creative.
BACKGROUND
The study of any area of Human-Computer Interaction encompasses a necessarily wide range of disciplines, such as computer and cognitive sciences, psychology, formal methods, art and design, and philosophy. Some work relates indirectly to informal interfaces, in that researchers have been experimenting with different concepts behind the human-computer interaction by using more intuitive graphical interfaces.
Sketching and
Informal Representations
In art and design there has been some work in analysing the principles behind sketching [5]. The authors explain how Leonardo da Vinci advocated the use of “untidy indeterminacies” for working out compositions, because he believed that sketches stimulated visual invention. Research from cognitive psychology [1] suggests that this is the case, in the way that a mental-imagery model is used by the human brain. It is suggested [16] that the brain can create a mental image of a sketch, and then apply processes to alter or enhance that image to useful and creative effect. Negroponte [14] notes that “Sketch recognition is as much a metaphor as fact. It is illustrative of an interest in those areas of design marked by vagary, inconsistency and ambiguity. While these characteristics are the anathema of algorithms, they are the essence of design”.
Lohse [12] indicates how research into cognitive models for the perception and understanding of graphs can be applied to informalism; rough-sketch representations of graphs are inherently interesting as informal objects. One of the key concepts from informal interfaces is the relaxation of the invariance of output by computers, so it is revealing to study how people perceive and process meaning from graphical information. Lohse describes a computer program UCIE (Understanding Cognitive Information Engineering), which models the underlying perceptual and cognitive processes used by people to decode information from a graph, and considers results from the analyses of bar charts, line graphs and tables. Lansdown [10] points out that computer graphics designers tend to aim at photographic realism when “convincing naturalism” might be more appropriate.
Dix [4] explores the concept of non determinism and informal reasoning in user interfaces, and proposes that deliberately introducing non determinism can sometimes ‘help’ in the system by actually reducing apparent non determinism for the user in a “limited non deterministic” system, i.e. one instance of non determinism can partially or fully cancel out another.
In his paper on Informalism in Interfaces [15], Reeker studies some examples of adaptive interfaces, and analyses concepts such as representations of visual knowledge, and projecting cognitive representational structures onto computational representations.
Cognitive
Dimensions
Green proposes the notion of “cognitive dimensions” [9] as a descriptive vocabulary to more accurately describe relevant interface qualities in cognitive rather than computational terms. He introduces notions such as viscosity (resistance to change), role-expressiveness and premature commitment. These concepts are further explored in [7] and [8].
Constraints and
Other Implementation Methods
Leler examines one central element of informal interface construction: the application of constraint satisfaction mechanisms [11]. This concept was used much earlier in Ivan Sutherland’s seminal work on the constraint-based graphical interactive system Sketchpad [18] and in Alan Borning’s ThingLab [2], as later expanded by the author [3] and others such as Stefik [17].
INFORMAL INTERFACE STRUCTURES
To put a conventional structure in place for informal interfaces, we can concentrate on the three key elements of the system - input, output and the internal representation. The example above assumed that the input device would be a stylus, drawing pen-ink on a screen (probably a hand-held LCD device), with output displayed directly on the LCD. Stylus input and tablet screen output is the closest technology at the moment to the natural and informal situation of drawing or sketching on a piece of paper.
The Input Mechanism
The input mechanism in the case of a stylus is as follows: a flow of pen-ink is input from the stylus position and both displayed on the screen and also stored in an internal pixel video buffer as a bit map or vector trace. Vector traces or bounding portions of the bit map buffer can them be passed to an informal object recogniser, which creates a set of frames [13], chosen from a database of prototype frames, by first extracting a prototype for each screen object. For instance, the object recogniser might first start to best-fit a straight line through the pixel group; if this was not successful it would try the next object in turn (a second order polynomial, perhaps) until it had exhausted its embedded list of possibilities or had found a match. An analysis would then be made of the residue, i.e. the difference between the actual pixel bit map and that forming the extracted prototype. This residue would be analysed, by working through a database of informal cognitive dimensions. For instance, the bit positions would be analysed for their perpendicular variance from the prototype to extract a measure of shakiness, and so forth for other required dimensions. Hence fillers for the slots of the particular frame are constructed.
The Output Mechanism
The output mechanism works by taking as input only the frame name (e.g. “straight line”) and filler values (e.g. values for shakiness, thickness and so forth) as parameters; the image builder takes this primitive and creates its image on the screen with its informal object drawing engine. So the recreated image (e.g. of a roughly-drawn straight line) may not be an exact copy of the original, but it is perceptually equivalent in that the objects it comprises will be recognised as the same objects as in the original, with the same variance.
The Internal Representation
A key element to the underlying architecture of an informal interface is the structure of the internal core, constructed using frames (as also detailed in other literature [6]), bound together by a purpose-designed spatial constraint satisfaction mechanism [11]. An operator from the input mechanism creates a change in a frame; the constraint satisfaction mechanism then propagates the changes and arbitrates constraints between all frames in the model; the resultant is then sent (again as an operator) to the output mechanism for redrawing. For instance, rotating a roughly-drawn square 45 degrees will result in another, perceptually equivalent, roughly-drawn square, although again the rotated image will not actually be an exact bit-map copy.
FUTURE RESEARCH
WORK
Current research is being focused on a number of areas, such as the construction of internal representational structures for informal objects, the decomposition and recomposition of such objects, and the application of functional operations (such as translation, rotation and addition) on informal objects. Software is being developed to experiment with these structures, and to allow for the examination of potential advantages and disadvantages for the users of such systems.
CONCLUSIONS
This paper has shown an outline of a new area of research in HCI, with the introduction of informality into interaction between human and computer. By coupling this notion of informality with internal frame and object representations and spatial constraint satisfaction mechanisms, we expect to be able to demonstrate novel computer interfaces which, while not necessarily appropriate for all interactions, may often allow for greater fluidity and expressiveness.
REFERENCES
1. Boden, M. The Creative Mind; Myths and Mechanisms. Basic Books, 1990.
2. Borning, A. ThingLab - A Constraint-Oriented Simulation Laboratory. Xerox PARC paper and Stanford Computer Science Department Report STAN-CS-79-746, 1979.
3. Borning, A., & Duisberg, R. Constraint-Based Tools for Building User Interfaces. ACM Transactions on Graphics, Vol. 5, No. 4, October 1986. pp 345-374.
4. Dix, A. Non Determinism as a Paradigm for Understanding the User Interface. Cambridge University Press. Paper in Formal Methods in Human-Computer Interaction. Harrison & Thimbleby (eds.) 1990.
5. Fish, J. & Scrivener, S.A.R. Amplifying the Mind’s Eye: Sketching and Visual Cognition. Leonardo, Vol 23, No. 1. pp 117-126, 1990.
6. Gonzalez, A.J. & Dankel, D.D. The Engineering of Knowledge-Based Systems - Theory and Practice. Prentice-Hall, 1993.
7. Green, T.R.G. The Cognitive Dimension of Viscosity: a Sticky Problem for HCI. In Human-Computer Interaction - INTERACT ’90. Diaper, D., Gilmore, G., Cockton, G. & Shackel, B. (eds.). Elsevier.
8. Green, T.R.G. Describing Information Artifacts with Cognitive Dimensions and Structure Maps. MRC Applied Psychology Unit. Paper in HCI ’91:Usability Now. Diaper, D. & Hammond, N.V. (eds.). Cambridge University Press.
9. Green, T.R.G. Comprehending and Manipulating Complex Information Structures. Queen Mary & Westfield College, London 1991. Summer School on “Theory & Methodology of Cognitive Science Applied to HCI Problems”.
10. Lansdown, J. Not Only Computing - Also Art. Computer Bulletin, Series III, 1 (Part 2), pp 18-19, 1985.
11. Leler, W. Constraint Programming Languages: Their Specification and Generation. Addison-Wesley, 1988.
12. Lohse, J. A Cognitive Model for the Perception and Understanding of Graphs. CHI ’91 Proceedings, ACM Press.
13. Minsky M. A Framework for Representing Knowledge. A.I. Memo 306, MIT Artificial Intelligence Laboratory, 1974.
14. Negroponte, N. On Being Creative with Computer Aided Design. Information Processing 77, I.F.I.P., Amsterdam, pp 695-704, 1977.
15. Reeker, L. Informalism in Interfaces. Paper presented at the Workshop on Informalism, Santa Cruz, California, May 28-30 1991. Incremental Systems Corp.
16. Scrivener, S.A.R. The Interactive Manipulation of Unstructured Images. International Journal of Man-Machine Studies (1982) 16, pp 301-313. Received 23 May 1981, revised 23 September 1981.
17. Stefik, M. Planning with Constraints. Artificial Intelligence 16 (1981), pp 111-140.
18. Sutherland, I.E. Sketchpad: A Man-Machine Graphical Communication System. Ph.D. thesis, MIT, Cambridge, Mass, 1963.