top of page

Interplayable Surface

An exploration on Augmented GUI that coexists with physical environments

Thesis Project


Hoon Yoon / Master of Industrial Design(MID) 2016 

Rhode Island School of Design(RISD)


Approved by Master's Examination Committee:

Andy Law

Graduate Program Director, RISD, Thesis Advisor

Timothy Maly

Writer, Design Researcher, Lecturer, RISD, Adjunct Thesis Advisor

Jemin Cho

User Interface Analyst, UX Designer, LG CNS, External Thesis Advisor

"P-GUI Sticker" for prototyping the augmented interface within the actual environment in the thesis process

Won the Gold Winner at the concealed category “Advanced Design & Design Research” of A Design Award 2017


The main goal of this experiment-driven thesis is to envision and design an interactive GUI(Graphic User Interface) that coexists with physical surfaces. Based on an understanding of user behavioral patterns for getting access to information in these types of situations, experimentations and prototypes are implemented and tested with participants. In particular, to observe the user behavioral pattern for augmented GUI within certain environments and circumstances, this thesis presents several types of participatory experimentations with physical GUIs. The experiment participants were encouraged to re-creat and reorganize physical GUI, relating to their own situational specificity or informational tendencies.

Based on extracted insights from research and experiments, in the last phase, I propose two thesis models about how interactive GUI applies to a physical environment: simulation mock-ups for user scenarios of augmented GUI and interactive GUI table combined with projection mapping.


Thesis Model 1: Simulative User Scenarios for Augmented GUI

Thesis Model 2: Interactive table reactive to user's haptic motion by projection mapping and pressure sensor

The thesis models will show several types of information structures and interactions related to people's behavioral patterns on augmented GUI. In framing the overall data structure and wireframe for the thesis product model, informative affordance corresponding with users’ situational specificity2 is considered a crucial direction point, actualized on an artifact in a perceptible way. Through experimentally prototyping a thesis model; consequently, I would like to expand the speculative usability interactive GUI will feature in the near future.



Interactive Table, HCI, GUI/UX, Augmented Reality, Affordance, Informative Hierarchy, Projection-Mapping, Site Specificity



Thesis Roadmap

Fundamentally, the thesis's ultimate direction point aims to speculatively design the augmented environment integrating with interactive, GUI-embedded artifacts via projection mapping. Based on experiments with the conditions and situations that digitize visual interactions coexisting with daily life, the thesis will experimentally propose the usability for how users interact with augmented information projected on an actual surface within a situational specificity.

To define the user for this thesis project, 

• The prospective user group for research and experiment in this thesis is partially limited to those skilled in or accustomed to using digital smart devices like smartphones, laptops, tablet PCs, etc.


• In terms of a user scenario, most of the thesis experimentations and outcomes are basically subject to the circumstance of hypothetically interacting with augmented GUIs within a certain situational specificity.

The flow chart below is about the overall sequential procedure for thesis activity as the main part of the whole thesis. 

 Figure 1: Sequential flow chart for planning thesis activity.

In the research phase, to analyze user’s usability pattern on information under situational specificity, 

• HCI framework will be utilized as the main methodology in Chapter 1.


• Analyzing the cases of a usability pattern into a consequential algorithm and individual and consolidated sequence models derived from HCI methodology will be utilized. 

• Based on data, insights will be clarified, relevant to constructing an informative structure for designing GUI at the final phase of the thesis.

In the experimental phase, in order to research users’ information propensity under situational specificity, participatory experimentations will be conducted with various types of participants to clarify user insights of informative tendencies in a certain situation.

In the prototyping phase:

• Physical GUI stickers will be used in Chapter 2 in order to simulate augmented reality in actual environments.

• In Chapter 3, participants will use a sticker pad to draw their own GUIs and attach what they draw on small-scaled LEGO interior sets representing average living spaces. 

• In Chapter 4, participants will use physical GUI stickers, attaching them to a certain spot in their private spaces like residences.

In the prototyping phase:

• Overall, projecting mapping will be utilized as the main tool for actualizing interactive GUI in a physical environment

To conduct visual projection mapping for experimentation, technical tools are suggested as follows:

• Processing will be mainly utilized for actualizing digital interaction and visualization as a way of programming code for it.

• Adobe Photoshop and Adobe After Effects will be utilized in order to prototype the scenario that the augmented GUI interplays with the physical environment.

• Based on the codings program from Processing, visual interaction will be projected on the surface via the projector. 

Thesis Activity Introduction

1. HCI framework on user propensity of informative accessibility

In this research phase, it aims to methodically analyze people’s informative propensity to get access to information at a certain situational specificity. In order to extract the relational attributes, people’s behavioral pattern indicates, the HCI framework(Individual Sequence Model, Consolidated Sequence Model) is utilized in clarifying the sequential algorithm of user’s informative tendency.

1) Basic Approach: Tendency of accessing information, depending on situational specificity

  Figure 2 : Several venues where we tend to get access to any kinds, any types of information 

(Clockwise from Top far left) 

1. Searching information with laptop on the table

2. Searching Music on the verge of washing dishes

3. Verifying bathroom sign on the verge of entering there

4. Watching news via smartphone while having lunch

5. Checking out recipe while cooking

6. Trying to check out school's time schedule on the desk, even though it is messed up by scattered clothes and some stuffs

7. Verifying modeling shop's available working hour

8. Watching blog via smartphone while waiting for a subway


2) Analyzing user’s inherent behavioral pattern with HCI methodology

Based on various types of activities of utilizing application via smartphone under situational specificity, I constructed several versions of the user’s behavioral sequence model in order to methodically analyze the algorithm of workflow that each events inherently contains. 

To introduce with examples, in Figures 3, user behavioral patterns for getting access to applications via smartphones under certain time conditions will be analyzed in a frame of HCI’s Individual Sequence Model. And then, in Figures 4, data will be reorganized and clarified into an activity definition, intention, sequence, and insight keyword in a frame of HCI’s Consolidated Sequence Model.

* The frames of sequence models showing from the following sections

are theoretically cited from the framework introduced in “Sequence

Model Analysis,” “Kim, Jin Woo (2012), Human Computer Interaction 개론

pp 254, 255.” On the basis of HCI frames, user behavioral patterns were

sequentially reorganized and clarified.

  Figure 3: Individual Sequence Model for a situation of getting access to a map application under time constraint.

  Figure 4: Consolidated Sequence Model for a situation of getting access to a map application under time constraint.

 2. Participatory Experiment(1)- Self-directed UI drawing

For practical usability, communicating with actual or expectable users and inherently getting to know about their preference pattern of approaching GUI in daily life are significantly important to comprise of the overall structure and contents within. From this phase, I started to think about the possibility that I encourage nearby people to engage in an experiment as a participant and give them a time to manipulate the graphical environment in their own way just as a playful activity. What were the most successful insights during this participatory experiment was that I was able to acquire their diverse, unexpected, even irrational approach of generating the idea for designing their own GUI on the small scaled-LEGO structure supposed as their living room in the experiment.

  Figure 5: Footages of conducting participatory experiment with my classmates and faculties of MID program, RISD

3. Participatory Experiment(2)-Physical GUI Prototyping & Personal preference survey

In designing the overall UI structure which consists of informative structure, speculatively measuring its adaptability of applicating it to the actual environment is crucial for configuring the functionality of what it is shown. In this chapter, to experimentally gauge the usability of UI structure that is anticipated to apply to the physical environment, I attempted to design several types of UI structure which could possibly be utilized in our daily lives, such as, navigating music application while or on the verge of taking a shower, verifying today’s weather forecast, checking the state of food in a refrigerator. In this phase, I contacted several participants in MID studio, RISD, and asked them to play with GUI stickers within their daily lives for approximately a week. Through this experimentation, I intended to see how participants re-organize and re-compose the sample GUIs into their own certain ways, which is deeply reflected in their daily lives and matters.

 Figure 6: Scenes of conducting an experiment: Peel off the GUI sticker, do the survey, attach it on the surface where participants feel to do so is a necessity.

 Figure 7: A scene of physical GUI prototyping by P-GUI Sticker_a situation of navigating the main page of augmented interface on the transparent window surface

 Figure 8: Footages of attaching physical GUI interface sticker(P-GUI Sticker) on the surfaces, based on situational and spatial specificity of accessing to certain information 

4. Experimental Assumptions on usability via Projection Mapping

In a situation of perceiving information from external stimulus comprised of informative-based interactions, people have different tendencies for acquiring it depending on situational, spatial specificities. In order to speculatively see how the graphic-informative interactions are adaptive to the external environment or circumstances, several types of visual projections containing certain contents were simulated as follows.

1) Pilot Demo: Small-Size Simulation via LEGO Interior

Prior to implementing experimentation, as a playful simulation, I utilized the LEGO interior set I used in the previous chapter and projected a simple movable GUI on the LEGO wall to speculate how graphic interaction applied to the physical environment.

 Figure 9: Small-scaled graphic interaction projecting on LEGO interior wall.

2) Simulating Augmented GUI in an Actual Environment

 Figure 10: Simulation Scenes of various types of augmented GUI that combines with actual environments 

5. Prototyping sequential GUI interaction on the surface

As an extension for actualizing augmented GUI combining with the physical surface, in this stage, I started to develop the concept into designing sequential interaction of GUI. I utilized Adobe After Effect for making the sequences of how Graphic motions are occurring and reacting in a certain way by tactile action, like physically tapping the surface. In simulating the prototyping, I deliberately acted as actually navigating application projecting on the surface in order to recognize the interactivity and affordance application’s GUI contains.

 Figure 11: Prototyping video of simulating the augmented GUI integrated with actual environments, in terms of practical usability 

6. Developing interactive surface combining with augmented GUI

1) Speculative Rendering

In envisioning how the information is effectively synchronized with the physical surface, I figured out the concept of a grid system for organizing and arraying information. Similar to the theory of a Web design’s grid system, in designing an information structure, I thought that utilizing a grid system would enhance the informative unity and consistency for users to understand its relational hierarchy and sequence between each information.

Prior to making an actual product model, I speculatively rendered two different versions of interactive surface-based tables and built rapid mock-ups.

 Figure 12: (Direction 1)Honeycomb-shaped interactive surface table in futuristic kitchen place

 Figure 13: An initial mock-up for Direction 1

 Figure 14: (Direction 2)An interactive surface comprises different shapes, sizes, and heights of grids

 Figure 15: An initial mock-up for Direction 2

Concept Critique & Direction Decision

On deciding a direction for designing an interactive surface that suitably interplays with the user, I shared lots of ideas and feedback with thesis advisors and people interested in what I was doing. Particularly for the method of arraying information and constructing information structure, the majority of people I met mentioned similar opinions. These were that Direction 2s various forms of grids could rather hinder the user’s attention on information, especially at a certain moment like quickly checking the weather forecast or schedule right before doing something. For perceiving information in real life, they expressed concerns that excessive information via virtual and physical elements happens at one time on the surface. In terms of actual usability, the grid needed to be fixed in a way that was more simplified, and the GUI structure needed to be more clarified and uniformed in order for the user to easily recognize it.

Given advisors and people’s advice, I have come to decide to refine and reinforce Direction 1s strategy. With maintaining the concept of a honeycomb-shaped grid as the main platform for containing GUIs, clarifying the informative sequence and structure combined with graphic interaction was continuously considered while developing the thesis model.

2) Preparation process for thesis prototyping

 Figure 16: A process of making a mock up for prototyping(sensor is yet to be embedded)

 Figure 17: A scene of making an initial version of the interactive table. To generate reactive graphic interactions on a surface, I concealed pressure sensors under each grid. This sensor will play the role of reading the user’s haptic stimulus and transmitting it to the computer.

 Figure 18: I continuously attached pressure sensors under the wood panel, connected each sensors to an Arduino’s breadboard

 Figure 19: This is the moment of simultaneously re-adjusting GUIs' coordinate while projecting it on a surface. To easily synchronize the graphic interaction with physical grids, I marked the sensor’s position as above by using post-it notes

 Figure 20: Footages before conducting small-scaled prototyping experiments with people 

3) Designing the interactive surface reactive to haptic gesture

Based on insights extracted from former research and observation, I promoted a small group session for watching the usability of what participants interact with by sensors and movable graphic motion. Basically, the content that consists of the interface was a simplified weather forecast GUI. I intentionally more focused on the Graphic motion part, synchronizing with pressure sensors concealed under the hexagonal grid. Because at the first stage of simulating the synchronizable graphic-motion with haptic action, I thought it would be more relatively crucial to see how it operates as a first step for developing reactive graphic motion-based coding. Through this usability test, I found several types of far-fetched graphic motions and earned valuable user feedback on my initial direction point.

 Figure 21: An usability test about interacting with augmented graphic interaction that co-exists with information

4) Expanding the Adaptability of Interactive Surface in an Actual Environment

Similar to the overall direction points of simplifying information, in this exercise, I also intended to map out simplified, pictogram-based information to the surface. Except for some contents for notification that require detailed information like news or schedules, most of the information is minimalized, and its affordance is clarified by emphasizing symbol texts.

(To explain, in order to make people recognize its functional attributes, I intentionally utilized emphasized, minimalistic typography instead of graphical pictogram icons)

 Figure 22: The honeycomb-shaped interactive surface that consists of informative  graphics(Time Table, News Feed, Music Icon, Scheduler, etc.)

 Figure 23: A footage of conducting small prototyping test at class 

 Figure 24: Several features of interactive surface by integrating with projection mapping

(Clockwise from Top far left) 

1. Virtually Scribbling on the surface

2. Augmented object recognition 

3.4.  Responsive graphic interaction by haptic gesture on the grid

7. Making a thesis model: Interactive Table integrated with augmented GUI

1) Thesis Show Preparation: Designing the whole experience of Thesis Show


Prior to RISD’s grad thesis show 2016, every participant made approximately 1/7 sized thesis model in order to adjust the overall site configuration of the show. The image below is what I made for my thesis model.

 Figure 25: Approximately 1/7 sized thesis artifact model for site configuration of thesis show 

 Figure 26: Other classmates trying to slightly re-adjust their artifacts' position, which are next to my artifact

 Figure 27: A footage of  arranging all the small-sized thesis models on the table 

2) Making ​the actual-sized grid table responsive to haptic gesture

In making the thesis model for thesis show, the overall interface and its information structure are expanded as well as the area of hexagonal grids is expanded. The haptic experience generated by touching the grid surface is also more developed in a way that its graphical response is relatively more diversified and interactive. And when it comes to the User Experience aspect for the interactive table, the pressure sensor inputs are increased as the array of grids is expanded. So the audience can experience more diversified graphic interactions coded with the sensor embedded in multiple grid spots. 



Images below are the process of making an interactive table structure and re-adjusting graphic interactions’ coordinate to each pressure sensor’s position.

 Figure 28: Process of making an interactive table. After done with making structures and arraying wood grids, I started to embed sensors and wires under the grids by drilling holes on the base plate

 Figure 29: A footage of re-adjusting the coordinates of graphic interactions according to pressure sensors' position

3) User Experience on interactive table

 Figure 30: Actual usability scenes of interactive table

4) User Experience with various audiences

 Figure 31: The interface was revised after thesis model review. Its structure was relatively simplified from the initial version, and the individual graphic interactions coded with embedded pressure sensors were more focused on individual response

 Figure 32: Footages of demonstrating thesis model in front of RISD Grads

 Figure 33: Individual graphical response by touch

 Figure 34: Audience experience at RISD Grad Show 2016


Hoon Yoon 

Master of Industrial Design(MID) 2016 

Rhode Island School of Design(RISD)



Approved by Master's Examination Committee:

Andy Law

Graduate Program Director, RISD, Thesis Advisor

Timothy Maly

Writer, Design Researcher, Lecturer, RISD, Adjunct Thesis Advisor

Jemin Cho

User Interface Analyst, UX Designer, LG CNS, External Thesis Advisor

"P-GUI Sticker" for prototyping the augmented interface within the actual environment in the thesis process

Won the Gold Winner at the concealed category “Advanced Design & Design Research” of A Design Award 2017

bottom of page