Design of an Interactive Experience with ... - ACM Digital Library

Design of an Interactive Experience with Medieval Illuminations: A Journey into the Beauty and Meaning of Medieval Portuguese Manuscripts ˜ ´ N. CORREIA, T. ROMAO, A. RICARDO, and T. MOTA, Departamento de Informatica and CITI, Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, Caparica, Portugal ˜ e Restauro and REQUIMTE-CQFB, M. J. MELO and R. CASTRO, Departamento de Conservaçao Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, Campus Caparica, 2829-516 Caparica, Portugal R. CARVALHO and A. MIRANDA, Instituto de Estudos Medievais and Instituto de História da Arte, Faculdade de Ciências Sociais e Humanas, Universidade Nova de Lisboa, Lisboa, Portugal This article presents the design, implementation, and evaluation of an interactive installation to explore color in medieval illuminations and their context. The main goal of the installation is to promote wide dissemination of Portuguese medieval illuminations as well as to achieve engagement of the public in the conservation of this treasured heritage. The interactive installation is based on the experience and results obtained in a series of hands-on workshops, where “real” colored paints were produced using techniques and pigments similar to the originals. The design and development process of the interactive installation was carried out by a multidisciplinary team in computer engineering, design, illustration, art history, history, chemistry, and conservation science. The installation, after several iterations on the content and technology, explores innovative computational interfaces and how they can be developed and used in cultural heritage. It has three components that work independently or as a full installation: (1) Virtual Scriptorium, (2) Interactive Panel, and (3) Augmented Book. The design rational, implementation, and evaluation of the modular installation are described, as well as how this work contributes to the wide dissemination of information on these extraordinary medieval manuscripts. Categories and Subject Descriptors: H 1.2 [User/Machine Systems]: Human Factors; H.5.2 [User Interfaces]: Input Devices and Strategies; J.5 [Computer Applications]: Arts and Humanities General Terms: Design, Experimentation, Human Factors Additional Key Words and Phrases: Interactive installation, medieval illuminations, tangible interfaces, pen and touch interaction, sensors ACM Reference Format: ˜ A. Ricardo, T. Mota, M. J. Melo, R. Castro, R. Carvalho, and A. Miranda. 2014. Design of an interactive N. Correia, T. Romao, experience with medieval illuminations: A journey into the beauty and meaning of medieval Portuguese manuscripts. ACM J. Comput. Cult. Herit. 7, 2, Article 13 (May 2014), 19 pages. DOI: http://dx.doi.org/10.1145/2626289 ˜ para a Ciência e a Tecnologia), Portugal, as part of the project entiThis research was funded by the FCT/MEC (Fundaçao tled “Color in medieval illuminated manuscripts: between beauty and meaning” (PTDC/EAT-EAT/104930/2008), CITI (PEstOE/EEI/UI0527/2011), and REQUIMTE (PEst-C/EQB/LA0006/2013), and Rita Castro’s PhD grant (SFRH/BD/76789/2011). ´ Author address: N. Correia, Departamento de Informatica and CITI, Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, 2829-516 Caparica, Portugal. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or [email protected]. c 2014 ACM 1556-4673/2014/05-ART13 $15.00 DOI: http://dx.doi.org/10.1145/2626289 ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13

13:2

1.

•

N. Correia et al.

INTRODUCTION

Research into medieval Portuguese illuminated manuscripts, many of which are masterpieces of medieval art, has opened a new interdisciplinary field, one that has led to new discoveries and reinforces the need to preserve this precious heritage [Melo et al. 2011; Correia et al. 2011]. During the course of this study, we came to understand how important it is to improve public awareness and access to this extraordinary resource. Consequently, since 2005, we have been promoting hands-on workshops, where participants paint with materials as similar as possible to the original ones including pigments, iron-gall ink, and parchment glue (Figure 1). This is a way of sharing the results of our studies and contributes to raise public interest. Unlike easel paintings, for example, medieval manuscripts are fragile tridimensional objects that cannot be fully accessed in current exhibitions. In a typical exhibition, only two pages from a whole book are put on display, and as a consequence much of the information and beauty cannot be seen and appreciated. This constraint may be overcome to some extent by using interactive technologies. To fulfill this goal, our team integrates contributions from art history, design, chemistry, conservation, and computer science. Manuscript illumination was one of the most important and original means of artistic expression in medieval Europe, and in Portugal’s case, provides the only painted works from the Romanesque period [Melo et al. 2011]. Our research into the color of medieval Portuguese illuminations revealed that these artworks also embody the most advanced science and technology of the time, a result of the coexistence of Arab, Jewish, and Christian cultures in this country [Melo et al. 2011; Melo et al. 2014]. Thus, these illuminated manuscripts should be perceived both as works of art and as technological achievements [Melo et al. 2011; Ball 2001]. The hands-on workshops we developed on the practice of medieval illumination are designed to bridge the more than eight centuries that separate us from the time of their creation. Experimenting with the ”real” natural sources of medieval colors, such as the beautiful minerals of lapis-lazuli, cinnabar, orpiment, or the roots of certain plants and parasitic insects (the raw materials for organic lake pigments), has proved to be fascinating for all: from small children to teenagers to the elderly. The discovery of colors and materials that are no longer available has been key to the workshops: participants can handle a parchment folio and prepare paint by grinding the pigments with a range of binders including parchment glue (a gelatin), glair (egg white), or gum Arabic (a plant resin). These activities, which introduce the art of painting a miniature, were the most exciting part of the workshops and created the perfect opportunity to form a bridge to the medieval world and to experience the beauty and meaning of the colors that were used to produce manuscript illuminations (for more details, see Section 3). Equally as important as the accurate information that we made available to the public is the sense of wonder that we wanted to preserve and re-create with the installation. For this reason, the design of the interactive installation has aimed to integrate all of the relevant visual and physical aspects of these precious and irreplaceable works of art as well as to introduce the context in which they were produced. Color is used as the core subject of discovery within three of the most important Portuguese Romanesque manuscript collections that were produced in the scriptoria ˜ Holy Cross of Coimbra and St. Mary of Alcobaça monasteries,1 during the of St. Mamede of Lorvao, th second half of the 12 and first quarter of the 13th centuries. Through this focus on color, access to these artworks can be greatly enhanced and their social and material context revealed [Melo et al. 2014; Brussatin 1983; Pastoureau 2000; Gage 2009]. These manuscript collections are presently kept in the Portuguese National Archive of Torre do Tombo ˜ (Lorvao), National Library of Portugal (Alcobaça), and Municipal Library of Porto (Holy Cross). 1 Portuguese monastery foundation dates: Sao ˜ Mamede de Lorvao ˜ (ca. 857), Santa Cruz de Coimbra (1131), and Santa Maria de Alcobaça (1153).

ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

Design of an Interactive Experience with Medieval Illuminations

•

13:3

Fig. 1. Hands-on workshop in the Science Museum in 2009 (left) and a reproduction of an illumination from one of the participants (right).

The modular interactive installation has three components that cover different aspects of medieval illuminations and uses several interactive techniques: (1) Virtual Scriptorium, (2) Interactive Panel, and (3) Augmented Book. The Virtual Scriptorium uses a digital calligraphic interface to enable the experience of producing illuminations. It provides background information about the illuminations being produced and is intended to be a single user experience. The results can then be shared in another component, the Interactive Panel. This panel, a large interactive multitouch surface, provides information about medieval manuscripts from a scientific, social, artistic, and historical point of view. It can be a shared experience, for example, used in a field trip class by students and instructors. Finally, the Augmented Book combines the physicality of the book, the codex, with digital materials related to how it was made and the beauty of its content. The structure of the article is as follows: Section 2 presents related work that helped to inform the design and development processes; Section 3 introduces the interactive installation and its relation with the hands-on workshops; Sections 4 through 6 describe in detail the individual components; Section 7 describes evaluations of the interactive installation by users; and finally, in Section 8, the conclusions and directions for future work are presented. 2.

RELATED WORK

Museums and cultural sites are increasingly augmented with digital technology, seeking innovative and entertaining ways to efficiently communicate and attract younger audiences. Interactive installations allow visitors to have an active role when exploring the available information [Correia et al. 2010]. They enable visitors to further control their path through the museum and information spaces and to experiment with virtual activities that simulate real processes that otherwise would not be possible. Hands-on exhibits provide engaging learning experiences, reaching all kinds of visitors regardless of age and interest profile [Hornecker and Stifter 2006]. Children and young people are especially captivated by interactive installations, quickly steering toward hand-on exhibits. Horn et al. [2009] compared the use of tangible interfaces and traditional graphical user interfaces and ascertained the benefits resulting from the application of the former for educational purposes. This kind of installation also plays a key role in history museums and art restoration, allowing visitors to access artwork without compromising its integrity [Bonanni et al. 2010]. Using an interactive system ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13:4

•

N. Correia et al.

[Flagg and Rehg 2006] to paint with traditional media tools allows a novice artist to focus on a series of subtasks in executing a complex painting. Several interactive systems to explore and disseminate illuminated manuscripts have been developed. Project Rerum Novarum allows the interactive exploration of illuminated manuscripts in digital libraries [Borghesani 2010a; Rerum Novarum 2013]. It focuses on the design and implementation of a new generation of multimedia systems for analysis, search, retrieval, and virtual navigation of the illuminated codex. Using multitouch interfaces, users can browse and search (text, images, and tags) a collection of manuscripts [Borghesani et al 2010b]. Turning the Pages [2013], developed by the British Library and Armadillo New Media Communications, supports the entire process for the creation of interactive book galleries for multitouch tables or the web. The application features a graphic engine to realistically represent manuscripts in 3D, allowing users to naturally turn a book’s pages. The Fitzwilliam Museum houses on its site a web application [The Making of a Medieval Manuscript 2013] that describes the process of creating an illuminated manuscript, from the making of the parchment to the writing and painting. Interaction with the application is done with the mouse. Users follow textual instructions and click on several points of interest to proceed or to trigger particular animations representing the different steps of the process. These applications focus on the display and exploration of the artworks or the description of their production process, but they do not allow users to try out those processes and virtually experience the creation of an illumination by themselves. Books have a familiar interface that has been unchanged for hundreds of years. Yet with the currently available technologies, it is possible to augment it in interesting ways, illustrating how a familiar interface can be enriched with different types of multimedia content. There are several types of augmented books that use sensors for different purposes. One approach is to detect when the pages are flipped, enhancing the experience by adding multimedia content. Listen Reader [Back et al. 2001] is a museum installation where music and sounds are used to enrich the experience of reading a book. As the users flip through the pages, the soundtrack changes while the story in the book unfolds. The sounds and music help to set the environment for the story. Electronic Popables [Qi and Buechley 2010] is an augmented pop-up book. As in any pop-up book, certain areas pop out on each page, but the pages are also interactive, responding to certain actions from the user. Clicking, dragging, and bending specific areas of the book trigger different outputs, such as light, sound, and even movement. This enables the creation of a more interactive and engaging experience. This work builds upon the effort of developing frameworks [Buechley et al. 2010] for the seamless integration of paper and computing. WikiTUI [Wu et al. 2008] is another model and toolkit departing from the traditional hypertext to extend it to the combination of printed books and additional media. Other augmented reality books where content is overlaid in the pages of a physical book have also been proposed. A mixed reality book for children [Grasset et al. 2008] was developed with the goal to assess the best ways to combine content and additional interactive information by considering the design space for this new medium. Educational content has also been used in augmented books, and ¨ the results show that the learning process improves [Dunser et al. 2012]. 3.

HANDS-ON WORKSHOP AND INTERACTIVE INSTALLATION

The hands-on workshops, generally, start with a historical and artistic context of Portuguese medieval illuminated manuscripts, followed by an introduction on the medieval materials and techniques that aims to engage the public in experimenting with the physical materials.2 Participants are invited to 2 Two videos were made during a workshop in the Science Museum (Pavilhao ˜ do Conhecimento) at Parque das Naço˜ es, Lisbon, May 2009: http://www.cvtv.pt/imagens/index.asp?id video=295 and http://www.cvtv.pt/imagens/index.asp?id video=302.



•

13:5

choose an illumination from the Book of Birds (1183–84) and the Apocalypse (1189) from the Monastery ˜ to be reproduced following the step-by-step instructions detailed in a worksheet (Figure 2). of Lorvao, First, the drawing is copied onto parchment; then, the paints are prepared, and, finally, the illumination is painted with the freshly made paints. We seek to use the materials of that time; however, toxic pigments are swapped for innocuous colorants when working with children and untrained people. For more information, please see Discovering Medieval Illuminations3 [2007]. Departing from the results of these workshops, the interactive installation allows one to replicate the experience of creating the illumination with the digital calligraphic interface in the Virtual Scriptorium. The Interactive Panel provides contextual information about the illuminations, and the Augmented Book combines the physicality of the codex with digital content about the medieval process of creating it. The next sections describe each of these components in more detail. 4.

VIRTUAL SCRIPTORIUM

This component aims to provide an individual experience of producing an illumination using a digital platform. As already described in Section 3, in the hands-on workshops, the challenge is to reproduce, using medieval materials and techniques, an illumination from the Book of Birds or Lorvao ˜ Apocalypse, ˜ this includes the production of paint both Romanesque manuscripts from the Monastery of Lorvao; color, based on medieval recipes and the scientific analysis of the manuscripts (Figure 3). The digital interface does not require the complex logistics and the several facilitators required to organize the “real” hands-on workshop. Another relevant aspect is the fact that in these hands-on workshops, some of the paints may require toxic pigments. Since the participants are encouraged to prepare paints for their own illuminations, this implies either not using those paints or having security measures that are hard to set up, especially when considering children. This application can be more easily placed in cultural sites with broad audiences, such as museums, public libraries, and archives, helping to divulge the art of illuminations through a playful and engaging experience. The Virtual Scriptorium, one of the digital components of the interactive installation, aims to support the construction of an illumination from the Book of Birds. Using a tablet PC and a pen instead of parchment and brushes, the user is guided through several steps in order to construct an illumination (Figure 3). Details such as the brush thickness or the running down of the paint in the pen remind of the difficulties and patient work involved in producing these illuminations. Most computer applications are designed with the goal of being as easy to use as possible. In this case, the user experience design is intended to provide the feeling of drawing a medieval illumination, and as a result, the process of creating it is as similar as possible to the one using real pigments. As in the real workshop, users can produce their own paints (Figure 3(b) and Figure 3(e)), through actions like grinding carefully a lapis-lazuli stone in order to have a proper blue pigment or mixing it with the binder; pigments like red vermilion should be thoroughly grinded in order to have a good color paint, but others like lapis-lazuli may turn grayish and lose their color if too grinded. The application helps participants discover this fact. It also contains details concerning each illumination from the Book of Birds, as well as the corresponding texts. Information on the birds depicted, and their symbolic meanings, is also provided (Figure 3(c)). The desired outcome is not to create an experience equivalent to a real workshop, as differences, such as the ones regarding touch and smell, are obvious, but to provide a new approach to the production of an illumination with its own characteristics maintaining a link to the traditional production process (Figure 4). More information on color may be explored with the Interactive Panel. 3 Original

` descoberta da iluminura medieval.” title in portuguese: “A ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13:6

•

N. Correia et al.

Fig. 2. Step-by-step worksheet on how to make the illumination of “The Dove and the Hawk,” from the Book of Birds. Key steps and instructions are detailed, starting with the making of the paints.



•

13:7

Fig. 3. Creating an illumination using the Virtual Scriptorium.

˜ Lv5, f.5, 1183– Fig. 4. Original illumination and digital version from ”The dove and the hawk,” Book of Birds, from Lorvao, ˜ Geral do Livro, dos 84. Lisbon, National Archive of Torre do Tombo (PT/TT/MSML/B/5). Reprinted with permission of Direçao Arquivos e das Bibliotecas. ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13:8

•

N. Correia et al.

4.1 Design, Structure, and Implementation Users are guided, step by step, in the process of producing the illuminations: drawing, preparing paints, and painting. In the proposed application, the production of illuminated manuscripts is done primarily through a tutorial that leads the users through a set of steps similar to the ones outlined earlier for the workshop. The main steps are as follows:

r r

r r

Selection of the illumination: Allows the user to see the illuminations available in the system and to select the one to reproduce. Production of the illumination: Allows the user to draw and add color to all the details of the chosen illumination, with the aid of a tutorial comprising the set of steps described earlier. During production, the user has access to paints to reproduce the chosen illuminated manuscripts, and although the user is instructed about which paint to use in every step of the tutorial, it is possible to choose a different one. In order to reproduce the real process, the users need to periodically dip the brush into the paint recipient. The user also has access to a set of tools for drawing and painting required to produce the chosen illumination, which are simulated through the automatic parameterization of the pen settings for each step of the tutorial. Obtain information about the illumination: Allows the user to obtain generic information and to navigate in a set of more detailed information about the illumination being produced. Paint preparation: Allows the user to experience the whole process involved in preparing paints, aided by a tutorial. In this tutorial window, the user has access to all tools, materials, and techniques needed to prepare the paint.

The application was developed using the Qt platform, an open-source development platform in C++ that supports the development of software with graphical interfaces. The content of the application is described in an XML file. As mentioned, the application runs on Windows tablet PCs with pressuresensitive pens. The visual design is inspired by the medieval manuscripts and specifically by the illuminations. The initial design of the interface suffered a series of changes throughout the development phase. These changes served primarily to improve the browsing interface and to better adapt the interaction style. The changes resulted from meetings with conservation and restoration experts and an early questionnaire-based user evaluation. Image display avoids using scrollbars and the pen is used to drag and move them. Kinetic scrolling was also added, as it is a very convenient method when only a pen is used. Visual feedback was added to the graphical elements in order to be clear how the user has impact on the system. Particular care was taken in the representation of objects such as the ceramic water container and the shells as paint containers. 4.2 Interaction The user interacts with a calligraphic interface on a tablet using a pen/stylus. This type of interface is the most natural choice for this application, taking into account that one of the main goals is to support drawing and writing. As the application relies on the expressiveness of drawing, the choice was to use a pressure-sensitive pen as input in order to capture the subtleties of the drawing process. This type of technology supports adjusting dynamically the line thickness and the amount of paint used by simply varying the pressure of the pen on the tablet surface, thus avoiding additional interface elements. The interaction is done only with the pen, and this influenced the design of the application especially considering the absence of a keyboard or mouse. Particular attention was given to a common problem in calligraphic interfaces, the fact that the user has to point in an absolute way, touching the desired locations with the tip of the pen. This entails an extra difficulty for interaction, taking more time for the ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.


•

13:9

Fig. 5. Color window in the Interactive Panel. The menu in this figure, with the several shells, enables the user to select any color paint. The circles below the red paint allow the user to access the following information, from left to right: “the color of a grain” (the physics and chemistry of pigments color), “how it is produced” (medieval recipes and rationale of medieval process to obtain vermilion), “how it is applied” in the manuscript, “motives and patterns,” and symbolic “meaning.”

user to perform each action. It is necessary to consider this constraint in the design phase, specifically the occlusion of some areas of the interface by the hand, the difficulty in pointing, and the need to use more complex operations, such as double click. To minimize these difficulties, the features of the application were organized in well-identified areas, in groups of functional sequences, with the most relevant features in places where occlusion is not a problem. The size of the buttons is large enough, requiring only one touch for action. The simulation of painting tools is performed automatically according to the steps of the tutorial. For each step, parameters associated with the pen are adjusted automatically, enabling the user to access all the necessary tools (e.g., nib pens and brushes of different sizes and shapes) without worrying about menus. To draw with any of these tools, the user has to tap and drag the stylus tip in the region dedicated to this purpose. Thanks to the detection of the pen pressure on the tablet surface, it is possible for the user to precisely control the amount of paint used and the thickness of the line. In order to select a paint, the user has to press the pen against the tablet, thus reproducing the original process of obtaining paint from a container. As mentioned, optionally, the user may prepare the paints. In the Virtual Scriptorium, the user is instructed when the perfect grinding is achieved (Figure 3(b)). This process is explained in the Interactive Panel (Figure 5). 5.

INTERACTIVE PANEL

The interactive touch panel provides an overview of the illuminated codex production process and its historical context. Special relevance is also given to color in medieval manuscripts from a scientific, social, artistic, and historical point of view. Thus, the panel’s main goal is to disseminate the discoveries of the research on color in medieval illuminations and contextualize it with historical information from the period being studied: the Middle Ages. This is achieved through the exploration of some of the more relevant medieval Portuguese illuminations. The digital exploration of these works contributes to their study but also to their preservation. The panel links directly with the Scriptorium by including a gallery of all the images/illuminations that were produced. ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13:10

•

N. Correia et al.

Fig. 6. Interactive Panel initial menu.

Since the application will mainly be used in cultural heritage sites, libraries, and museums, the interface and content were designed for a large multitouch surface that could be used by a group of visitors. This component may be deployed alone or in conjunction with the other applications that were developed. The design considered the requirements of having a simple-to-use, informative, and playful interface that would target a heterogeneous audience consisting of users of different ages and skill levels. As mentioned, the interaction with the proposed application is made through a touch interface. Users interact with the system through the touch of a finger on its surface and may use one or both hands. This interaction also includes the execution of some specific gestures to perform certain actions in particular areas of the interface. Most interactions with the system, for navigation and exploration of information, are done by touching in well-defined areas, including buttons and icons representing the topics to explore. The interface design went through several iterations during the development process. The changes were especially aimed at improving the organization of the provided information, adapting information to the interaction modes, and simplifying the navigation in the application interface. The interface design is inspired by the explored theme, with some decorative elements of the illuminations being reused in the visual elements of the interface. This aims to further engage the user. The navigation interface has a hierarchic structure departing from the initial menu (Figure 6), where each node corresponds to a specific topic. Given the possible casual use of the interface, some users might not return to the initial menu and thus an automatic return transition occurs after 5 minutes of inactivity. Starting from the initial menu/panel inspired by medieval genealogical diagrams, users may explore several items such as Colors, Scriptorium, Codex, or Romanesque Portugal (Figure 5 and Figure 6). Besides the physics and chemistry of pigments color, the item Colors explores the way paints were produced in the Middle Ages. It also shows how they were applied in the codex and which are the most representative patterns and combinations with other colors. Finally, it also explores their symbolic and social meanings. ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.


•

13:11

Fig. 7. Scriptorium window in the Interactive Panel.

The item Codex also plays an important role, showing the enormous importance of the book at that time (in spite of its limited access) and displaying how the page layout of an illuminated manuscript was carefully designed regarding its different functions. It displays virtual versions of the codices Book of Birds and Lorvao ˜ Apocalypse in full size that the user may flip as if a real codex. Since the monks were the agents of codex production at that time, the item Monastery provides an overview of the site where the workshops or scriptoria were placed, showing also a glimpse of a monk’s life during the Middle Ages. In the Scriptorium (Figure 7), the user can explore the instruments and materials used by the copyist or by the illuminator during the process of production of the codex and also acknowledge the different tasks involved. By touching certain elements, additional information about those elements and the overall production process is provided. Romanesque Portugal is an item in which the user can find summarized information about the context of the Iberian Peninsula between the second half of the 12th century and the first half of the 13th century, namely, the multicultural Portuguese society, its organization, and the art that was produced. The panel also contains a gallery of illuminations that were previously made with the Scriptorium application. The application was developed using openFrameworks, an open-source C++ library, suitable for the development of creative applications, with a strong visual component. The panel content is described and organized in several XML files. Regarding hardware, the application runs in any configuration using the TUIO [Kaltenbrunner 2005] protocol for transmitting multitouch events detected in an interactive surface. We are using a touch panel manufactured by Samsung (LCD screen with infrared touch detection), but other devices using this protocol could also be used. 6.

AUGMENTED BOOK

Our approach for the Augmented Book is different from the ones described in the literature, in Section 2 [Borghesani 2010a; Rerum Novarum 2013]. Our goal is to maintain the form factor, weight, and binding method of medieval books and integrate the sensors and augmented behavior as seamlessly as possible. This allows for assessing all the physical aspects of the medieval books while adding digital content. ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13:12

•

N. Correia et al.

Fig. 8. The Augmented Book.

The Augmented Book (Figure 8) is the most exploratory component of the installation regarding the interface device. In opposition to the other two components, it provides a physical experience of the book, while the digital content augments this experience. All of the electronics are hidden within the wooden covers of the book, to make the book as similar to a real codex as possible. For this purpose, we used the same type of binding found in the Romanesque book bindings from the Alcobaça ˜ Monastery. Users can peruse a Monastery, with the same size of the Book of Birds from the Lorvao real codex and look at painted illuminated illustrations from the Book of Birds. The pages with the illuminations and the covers contain sensors that detect the orientation of the book and the page position. At the same time, the model of the same book is displayed and projected in a large screen accompanied with information relating to its binding, describing the materials, actions, and elements necessary to construct a book and make it a functional object. For instance, as the user turns the book, facing its spine, visual and written information is provided about the way quires were sewed. Information concerning the parchment and some relevant elements of the folios is also accessible through actions like opening the book or turning pages. The book can detect when some specific pages are open, enabling the display of virtual content related to that page. Within each page, it is also possible to navigate through the content by performing swipe gestures with the hand in front of the book (using the distance sensors in the cover). All of these actions can be performed whether the book is standing on a table or is being held by the user. The augmented content application was programmed with openFrameworks. The Arduino controller was used to gather data from the sensors. The book uses capacitive sensors for detection of specific pages, distance sensors for virtual page flip by detecting gestures, and an inertial measurement unit for position and motion, allowing the user to explore the book in several orientations. It also includes a wireless communication module to communicate with the desktop application that processes the data and presents the augmented content.

7.

USER STUDIES

User studies were planned and conducted to evaluate the installation usability, analyze the users’ reactions, and provide insights for further developments. The following sections describe these studies and the results that were obtained. ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.


•

13:13

Table I. Main Statements Included in the Questionnaire (Second and Third Part) 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Drawing The instructions are easy to understand The tools are easy to identify The need to dip the brush is natural Pigment preparation The materials are easy to identify The instructions are easy to understand Dragging the materials to the mortar is easy Grinding the materials is easy Illuminations information Texts are useful to understand the meaning of the illuminations It is easy to understand the texts General aspects The difficulty of producing illuminations is correctly reproduced by the application The information about the illuminations is very useful I understood the whole illuminations’ production process I would recommend this application Overall satisfaction

7.1 Virtual Scriptorium The Virtual Scriptorium was tested in two different situations as described here. It was first deployed during a large event for teenagers (ExpoFCT2013), an open day at our university campus attended by 5,000 high school students. A second evaluation session was performed during the Medieval Europe in Motion International Conference held at the National Library in Lisbon.4 During ExpoFCT, held in April 2013, visitors were free to experiment the numerous available activities including the Virtual Scriptorium. The tests were performed on a population of 29 voluntary participants (45% male and 55% female), ranging in age from 15 to 25 years old with a mean age of 17.6 years; 90% of the participants claim to be familiar with new technologies, but only 59% say they had previous contact with calligraphic interfaces. The test sessions were conducted individually for each user by one researcher. Before starting to use the application, users were informed about the application and the objectives of the test. The users were encouraged to “think aloud” and tell what was going through their minds while using the application. Afterward, participants were asked to fill out a questionnaire. During the tests, the researcher took notes of the users’ behaviors and verbalizations, providing a record of any issues explicitly mentioned by users. The questionnaire comprised questions regarding the participants’ personal data and experimental feedback, as well as suggestions and comments. Personal data included the participant’s age, gender, and familiarity with new technologies. Experimental feedback was evaluated through three different sections of the questionnaire comprising several statements each. The first section contained general usability aspects, the second focused on the three main features of the application (drawing, pigment preparation, illuminations’ information and selection), and the third was related to general aspects of the interface. Users had to indicate their level of agreement with each statement by circling a value on a 5-point Likert-type scale, with a response of 1 meaning “totally disagree” and a response of 5 meaning “totally agree.” Table I shows the statements included in the questionnaire. Most participants agree or totally agree that it is easy to interact with the application (76%); the instructions made available are adequate for them to find their way around (90%) and the content 4 https://sites.google.com/site/medievaleuropeinmotion2013/.


13:14

•

N. Correia et al.

Fig. 9. Virtual Scriptorium: Users’ experimental feedback (first evaluation session).

is well organized (90%). The results from the second and the third parts of the questionnaire are displayed in Figure 9 and described later. The second part of the questionnaire was divided into three groups of questions, each one related to one of the main application features:

r

r r

Illuminations drawing and painting: The instructions provided by the application on how to draw an illumination were easy to understand by most users and users could easily identify the available tools by the corresponding icon. However, some improvements can still be done to clarify the instructions and the icon design (e.g., there is no clear distinction between the black ink and the black iron gall). To replicate the real process, while painting an illumination the users need to periodically dip the brush into the paint recipient. It was interesting to see some users surprised when the brush gradually stopped painting. Some had to pause and think, but they quickly figured out what was happening. People generally suppose that digital tools overcome the problems of the manual tools, so they were not expecting to face this traditional problem. Yet most of them considered it a natural process. Since the current version of the Virtual Scriptorium is not directed to professionals but to the general public, we meant to reproduce the ancestral process as accurately as possible. The majority of the participants could easily access the information regarding the illuminations and proceed to the pigment preparation; however, the process could even be easier if the icons for each step are highlighted. Pigment preparation: The vast majority of respondents agreed that the instructions were easy to understand and the materials were easy to identify, and even highlighted the ease of dragging and grinding the materials in the mortar. Illuminations information: Most participants considered the texts easy to understand and useful. They also agreed that the design was appealing.

Concerning more general aspects, participants agreed that the difficulty of producing the illuminations is correctly reproduced by the application and considered that they understood the whole illumination production process. Participants stated that the provided information regarding the illuminations is useful and they would recommend the application to their friends. Overall, the participants’ satisfaction was very positive. There were comments on the need to improve the pen calibration, which is a key element of the interface. This drawback was mainly noticed while drawing the illuminations’ outline and it was promptly considered. During the Medieval Europe in Motion International Conference held at the National Library in Lisbon, additional tests were performed with professionals from related research areas, such as art ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.


•

13:15

Fig. 10. Virtual Scriptorium: Users’ experimental feedback (second evaluation session).

history. The tests were performed on a population of eight voluntary participants (50% male and 50% female) ranging in age from 21 to 66 years old with a mean age of 42.2 years; 87.5% of the participants claim to be familiar with new technologies, but only 37.5% say they had previous contact with calligraphic interfaces. We followed the same methodology used during the previous user studies and the participants filled out the same questionnaire. Similarly to the previous evaluation session, all participants agree or totally agree that it is easy to interact with the application; most of them also agree or totally agree that the instructions made available are adequate to find their way around (88%) and the content is well organized (100%). The results from the second and third parts of the questionnaire are displayed in Figure 10. The results from this second evaluation confirmed the results obtained in the previous one. Regarding the questionnaire, the participants’ feedback was similar but even more positive, as can be seen from Figures 9 and 10. The users were really impressed and enthusiastic with the application, and in their opinion, it properly reproduces the traditional process and the skills required to accomplish it. Along with some compliments regarding the application, participants made several suggestions. They would like to be able to select different line thicknesses for outlining and painting the illuminations and to have more illuminations available for selection. They also suggested the possibility of zooming the illuminations to facilitate the visualization, but they understood that it would not allow a faithful reproduction of the traditional process (illuminations should keep their original size). They foresaw the use of the Virtual Scriptorium as an appropriate learning tool for children. Moreover, they proposed that, during the pigment preparation, users could visualize the creation of the colors in more detail (i.e., the color would change according to the quantity of water and pigment added to the recipient). Discussion of Results. Participants in the first evaluation session were teenagers and young adults, which represent a group of target users with little knowledge about medieval illuminations. In the second evaluation session, we aimed to collect the opinions of professionals working in medieval art studies and history-related areas, since they have different expertise and motivation. Results from both evaluation sessions were very encouraging. Participants from the second evaluation session were even more enthusiastic, since they could anticipate the application’s usefulness to increase people’s cultural awareness and appreciate the connection between the old and the new, the manual and the digital process. It was important to notice that in their opinion, the Virtual Scriptorium is able to faithfully replicate the traditional process. ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13:16

•

N. Correia et al.

Fig. 11. Interactive Panel: Users’ experimental feedback.

Results indicate that tools like the Virtual Scriptorium are able to provide users with a rewarding and fun experience, allowing them to acquire knowledge and practice activities that would not be possible in the traditional way for several reasons (i.e., materials would not be available for everyone; some materials could be dangerous to manipulate, expensive, or not available anymore). Certainly, the users’ sensitivity and accuracy when using the pen as a brush are still noticeably different from the real process, since they are not using real ink that can speckle the manuscript and they cannot feel the manuscript texture (they can visualize it). However, the Virtual Scriptorium seems able to easily make users understand the medieval illuminations’ production process. Several improvements were suggested. When painting an illumination, it is not clear how to proceed to the preparation of the pigments (which button should be used). During the pigment preparation process, some users do not notice the need to use the pen to mix the colors, and this must be better conveyed. The pen thickness was also mentioned as an issue that must be considered in the light of the users’ opinions mentioned earlier. 7.2 Interactive Panel Like the Virtual Scriptorium, the Interactive Panel was also evaluated by professionals from medieval art and history-related areas during the Medieval Europe in Motion International Conference held in the National Library in Lisbon. The tests were performed on a population of 11 voluntary participants (45.5% male and 54.5% female) ranging in age from 20 to 59 years old with a mean age of 37.5 years. All participants claim to be familiar with new technologies and to have previous experience with touchscreen interfaces, related with the use of smartphones and tablets (45.5% have already used an interactive table). The same methodology used for the Virtual Scriptorium user studies was followed. The questionnaire comprised questions regarding the participants’ personal data and experimental feedback, as well as suggestions and comments. The answers to the questionnaire were evaluated on a 5-point Likert scale from 1 (“totally disagree”) to 7 (“totally agree”). All participants agree or totally agree that it is easy to interact with the application; the instructions made available are appropriate, the content is well organized, and the clicking (touching) is a natural interaction technique. Most users also consider that dragging interface elements is a natural technique to interact with the application (82%). The middle part of the questionnaire was divided into several groups of questions, each related to one of the main application’s features (Figure 11): ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.


r r r r

r r

•

13:17

Colors: The participants considered that the information on the various aspects related with the color was easy to understand: origin of colors (Q1), color creation process (Q2), color meaning (Q3), color usage in manuscripts (Q4), and common color combinations (Q5). Gallery: Users liked the illuminations’ layout (Q6) and they considered the illuminations’ size appropriate (Q7). It was easy for them to use the kinetic scroll to browse the illuminations (Q8). Scriptorium: Participants considered that the steps to create a book were easy to understand (Q9). Book of Birds: The participants appreciated having digital versions of the codices available for visualization and browsing. For most participants, it was easy to turn the virtual books’ pages by pressing and dragging the bottom corners of the pages with their fingers as if they were flipping the pages of a real book (Q10). For some participants, the sound played during page flipping was relevant to make the process more natural and real, but it didn’t seem crucial (Q11). Romanesque Portugal: Users were also satisfied with this part of the application and they thought this item was important to convey the proper historical context (Q12). Codex: Users stated the navigation through the codex was easy (Q13).

Concerning more general aspects, participants agreed that the various concepts were clearly presented (Q14) and they would also recommend the application to their friends (Q15). Overall, the participants’ satisfaction was very positive (Q16). Participants also made some comments regarding the application’s content and interaction techniques. They suggested some details they considered interesting to include in the texts and they proposed the development of a zooming tool that allows the analysis of the images in more detail. Overall, the feedback was very encouraging and our observations revealed that users were very excited about the interactive panel and especially the interaction techniques. They easily understood the interaction mechanisms. Although they were standing and using their arms (hands and fingers) to interact, they did not seem to become tired and never complained (we need to consider that the Interactive Panel is directed to casual interactions). According to the users, the text legibility needs to be improved. Although the users seem not to give much importance to the sound played when they turn the pages of the codices, it is an important feedback element that they could miss once we remove it. 8.

CONCLUSIONS AND FUTURE WORK

The installation for exploring medieval illuminations and, in more depth, all aspects related with color, such as the symbolic and historical ones, was designed and developed by a multidisciplinary research team from areas including computer engineering, design, illustration, art history, chemistry, and conservation science. Initial requirements and sketches were done in collaboration with experts that run workshops on the production of medieval illuminations using, as much as possible, the original materials. The installation further extends this concept by allowing a wider dissemination of the beauty and technological achievements of medieval illuminations. Interactive technology is easier to deploy, avoids the problem of manipulating frail medieval manuscripts, and may attract new publics and interested participants. The three components that were designed and developed had several iterations and were reviewed by the art historians and conservation science experts on the team. Prototypes were evaluated during development, and both the Virtual Scriptorium and the Interactive Panel were evaluated in the final versions with diverse groups of users as reported in this article. Results indicate that tools like these ones are potentially able to provide users with a rewarding and fun experience, allowing them to acquire knowledge and practice activities that would not be possible in the traditional way. Both experts and the general public surveyed the Virtual Scriptorium. They were all very enthusiastic, and the experts considered that it faithfully reproduces the original illuminations’ production process. Overall, the users’ feedback was very encouraging, and our observations ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.

13:18

•

N. Correia et al.

revealed that users were very thrilled about the interactive installation, easily becoming familiar with the interaction mechanisms. Regarding future work, we still have to conduct a more thorough evaluation of the Augmented Book. This component had preliminary evaluations regarding usability of the sensor system, but a more complete evaluation is necessary. Furthermore, we are considering supporting other platforms (e.g., Android), especially for the Virtual Scriptorium component. ACKNOWLEDGMENTS

The original manuscript illuminations are preserved in Portuguese archives, and the research that we have carried out has been made possible by the generous support of the directors and staff from the National Archive of Torre do Tombo, Municipal Library of Porto, and National Library of Portugal. We would like to thank, particularly, the director from Torre do Tombo, Dr. Silvestre Lacerda, for his support with the interactive installation project. We would like to thank Jose Arjona and Pedro Oliveira for their help in the evaluation process and results analysis. And Leslie Carlyle for her text editing and helpful discussions. REFERENCES M. Back, J. Cohen, R. Gold, S. Harrison, and S. Minneman. 2001. Listen reader: An electronically augmented paper-based book. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’01). 23–29. P. Ball. 2001. Bright Earth: Art and the Invention of Color. University of Chicago Press, Chicago. L. Bonanni, M. Seracini, X. Xiao, M. Hockenberry, B. Costanzo, A. Shum, R. Teil, A. Speranza, and H. Ishii. 2010. Tangible interfaces for art restoration. International Journal of Creative Interfaces and Computer Graphics. 1, 1, 54–66. D. Borghesani, C. Costantino Grana, and R. Cucchiara. 2010a. Rerum novarum: Interactive exploration of illuminated manuscripts. In Proceedings of the International Conference on Multimedia (MM’10). 1621–1624. D. Borghesani, C. Grana, and R. Cucchiara. 2010b. Surfing on artistic documents with visually assisted tagging. In Proceedings of the International Conference on Multimedia (MM’10). 1343–1352. M. Brussatin. 1983. Storia dei colori. Giulio Einaudi, Torino. L. Buechley, S. Hendrix, and M. Eisenberg. 2009. Paints, paper, and programs: first steps toward the computational sketchbook. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (TEI’09). 9–12. N. Correia, T. Mota, R. Carvalho, and A. Ricardo. 2011. Interactive technology to explore medieval illuminations. Revista de História da Arte FCSH-UNL série W, 1, 294–299. N. Correia, T. Mota, R. Nóbrega, L. Silva, and A. Almeida. 2010. A multi-touch tabletop for robust multimedia interaction in museums. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS’10). 117–120. Discovering Medieval Illuminations. Available at http://www.dcr.fct.unl.pt/arquivo-digital/luminura-medieval. ¨ A. Dunser, L. Walker, H. Horner, and D. Bentall. 2012. Creating interactive physics education books with augmented reality. In Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI’12). 107–114. M. Flagg and J. Rehg. 2006. Projector-guided painting. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST’06). 235–244. J. Gage. 2009. Colour and Culture. Thames and Hudson, London. ¨ R. Grasset, A. Dunser, and M. Billinghurst. 2008. Edutainment with a mixed reality book: A visually augmented illustrative childrens’ book. In Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology (ACE’08). 292–295. M. Horn, E. T. Solovey, R. J. Crouser, and R. Jacob. 2009. Comparing the use of tangible and graphical programming languages for informal science education. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI’09). 975–984. E. Hornecker and M. Stifter. 2006. Learning from interactive museum installations about interaction design for public settings. In Proceedings of 20th Conference on Computer-Human Interaction (OZCHI’06). 135–142. M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza. 2005. TUIO—A protocol for table-top tangible user interfaces. In Proceedings of the 6th International Workshop on Gesture in Human-Computer Interaction and Simulation (GW’05). M. J. Melo, R. Castro, and A. Miranda. 2014. Colour in medieval portuguese manuscripts: Between beauty and meaning. In Science and Art: the painting surface. A. Sgamellotti, B. G. Brunetti, C. Miliani (Eds.), Royal Society of Chemistry, London. ACM Journal on Computing and Cultural Heritage, Vol. 7, No. 2, Article 13, Publication date: May 2014.


•

13:19

M. J. Melo, A. Miranda, C. Miguel, R. Castro, A. Lemos, V. S. F. Muralha, J. A. Lopes, and A. Gonçalves. 2011. The colour of medieval Portuguese illumination: An interdisciplinary approach. Revista de História da Arte FCSH-UNL, série W, 1, 152– 173. Available at http://revistadehistoriadaarte.wordpress.com/. M. Pastoureau. 1997. Jésus chez le teinturier. Couleurs et teintures dans l’Occident médiéval. Le Léopard d’Or. Paris. J. Qi and L. Buechley. 2010. Electronic popables: Exploring paper-based computing through an interactive pop-up book. In Proceedings of the 4th International Conference on Tangible, Embedded, and Embodied interaction (TEI’10). 121–128. Rerum Novarum. 2013. Homepage. Available at http://imagelab.ing.unimore.it/rerumnovarum/RerumNovarumHome/ Rerum Novarum.html. The Making of a Medieval Manuscript. 2013. Available at http://www.fitzmuseum.cam.ac.uk/pharos/images/swf/manuscript/ manuscript 5a.html. Turning the Pages. 2013. Homepage. Available at http://www.turningthepages.com. C. Wu, S. Robinson, and A. Mazalek. 2008. Turning a page on the digital annotation of physical books. In Proceedings of the 2nd International Conference on Tangible and Imbedded Interaction (TEI’08). 109–116. Received July 2013; revised December 2013; accepted February 2014
