Touch-Centered Interaction Paradigms for Time-Oriented Data

(1)

Touch the Time:

Touch-Centered Interaction Paradigms for Time-Oriented Data

P. Riehmann¹, J. Reibert¹, J. Opolka², and B. Froehlich¹

1Bauhaus-Universität Weimar, Germany

2Fraunhofer IAIS, Sankt Augustin, Germany

Abstract

We present novel direct-touch centered paradigms for exploring time series data visualized in a coordinated multi-view display.

The views can be easily and flexibly configured by choosing from various adapted or enhanced time-oriented visualizations. A vocabulary of robust direct-touch gestures enables the user to create, filter, derive, and focus on subsets of time-oriented data within the temporal dimension as well as the (data) records dimension. Our two-dimensional Focus&Context technique works particularly well with our extension to horizon graphs, which enables interactive scaling from area charts to multiply folded Horizon Graphs down to pixel-based visualizations and back.

Categories and Subject Descriptors(according to ACM CCS): H.5.2 [Computer Graphics]: User Interfaces—Graphical user interfaces

1. Introduction

Analyzing time series data sets consists mostly of a set of recurring basic tasks such as defining ranges, comparing ranges within or across time series, exploring a temporal range at higher resolution, or creating subranges in time or subsets of time series. Our work was motivated by the observation that multitouch displays allow the definition and manipulation of a range by simply using two fingers of the same or different hands. Thus, we developed a framework for research on touch-centered paradigms for visualizing and interacting with time-oriented data. Multiple time series can be shown with a set of different time-oriented visualizations in a coordinated multi-view layout. Since well known interfaces using touch displays such as iOS or Android usually do not follow the conventional windowing paradigm but rather present their apps via full screen or splitting it for two apps; we also follow this approach but support a flexible and recursive tiling to take advantage of larger screens. We contribute the following touch interactions for exploring time series datasets:

• A vocabulary of touch gestures (altogether referred to asSegment and Zoom) in order to create, filter, and derive sets and subsets of time-oriented data within the temporal dimension, as well as in the (data) records dimension.

• An interactive extension of Horizon Graphs [RP08] referred to as Pinch and Foldfor appropriately folding and scaling entire sets, as well as individual Horizon Graphs, which serves as a multi-focalFocus&Contexttechnique.

• A seamless way to transform regular Horizon Graphs into pixel- based representations and back again.

• United Focus as a two-dimensional Focus&Context approach combining the focus on a subset of the time series with an orthogonally-defined focus region along the time axis.

Our video [RROF18] demonstrates the fluent analysis of time series datasets with our multitouch gestures.

2. Related Work

The number of publications focusing on time-oriented data is im- mense; however a useful entry point and overview is Aigner et al.’s book [AMST11]. Another good source is Keim’s work on pixel- oriented visualization techniques [Kei00]. In this work he clas- sifies the different techniques at hand to develop pixel-oriented visualizations and formalizes their design process. Circle Seg- ments [AKK96] is a pixel-based, time-oriented visualization technique presenting multiple dimensions of a dataset where the discrete time steps are mapped to individual pixel colors from the center outward. Circle View [KSS04] is somewhat similar to the previous technique but time-spanned data is binned and visualized as an area. The aggregation allows the presentation of an arbitrary amount of data in the given screen space but lacks information about an individual datum. However, circle-based techniques can- not use only a fraction of the pixels on today’s rectangular 16:9 aspect ratio screens. Our system follows a more obvious layout de- picting the temporal dimension horizontally and the record dimension vertically, which can make use of the entire resolution of a

c

2018 The Author(s)

Eurographics Proceedings c2018 The Eurographics Association.

(2)

display. Hierarchical Temporal Patterns [LAB^∗09] use a calendar- like matrix interface to present the data in a hierarchical fashion and thus visualizes the data at different time granularities at the same time. Again, the data for a specific time-span is aggregated and represented as an area in a calendar-like interface allowing an identification of recurring patterns in the data.

Horizon Graphs were first developed as a space efficient method to visualize stock data [RP08]. The technique uses a special kind of area charts for each individual stock. In order to preserve screen space, each graph is vertically partitioned in discrete bands colored with increasing saturation. Each range is then layered on top of the band previously being located directly below. Negative values are flipped and drawn in a different hue. While the interpretation of horizon graphs has to be learned, they can be more effective than standard line plots when the chart sizes get quite small [HKA09].

Perin et al. [PVF13] further extended horizon graphs by allowing the user to adjust the baseline for each graph interactively. They also introduced an interactive zoom for values. The zoom does not change the height of each chart whereas our approach increases or decreases the folding of the Horizon Graphs.

ThemeRiver [HHN00] follows a different approach and uses stacked graphs around a centered x-axis to visualize quantity values over time. A very nice improvement of this technique was pre- sented by Byron and Wattenberg [BW08] by emphasizing aesthetics to bring this kind of visualization to a mass audience. With TouchWave [BLC12] Baur et al. extended stacked graphs by in- troducing touch-based interaction techniques. They provide a flexible way to query specific values and aid in the vertical comparison between different items. They also focus on scalability, especially for smaller touch-based devices. Another addition is the introduction of hierarchies within the stacked graph to hide some of the complexity. A more general and formal approach to multi-touch gestures was developed by Kammer et al. [KWK^∗10]. They developed a grammar and a test framework to investigate the ease of use of their proposed gesture formalization. Our approach focuses on a minimal set of touch gestures with one, two, or four touchpoints (two per hand) that work well with our screen tiling approach and time-oriented data.

3. Visual Concept

Our approach of arranging and dividing the working space resem- bles some aspects of tiling window managers preferred by many professional UNIX/Linux admins and developers challenged with multiple terminals to track and to interact with. We also aimed at preserving screen space and avoiding occlusions, as well, by recursively tiling so each pixel of the full resolution is used for display- ing information at every step during interaction. This is especially intended for group situations such as presentations or analytics ses- sions with a leading analyst operating the system. Interacting with a display with bare hands is an advantage, since arranging views with a classic window system may interrupt the flow of a presentation. For this reason we also do not provide scrollbars. Instead, we employ approaches that adapt and extend Focus&Context techniques.

When starting our visualization system, the workspace is empty

Figure 1:Splitting the screen by adding a new view of the selected dataset showing a different visualization technique. While dragging the new view across the existing views the system provides instant feed forward about the tiling possibilities and where the new view is going to be positioned after dropping it (upper image). The lower image shows the result.

and any of the datasets shown in the right part of the top bar can be selected. Any of the visualizations depicted in the left part of the top bar can simply be dragged into the workspace filling then the entire space available. Additional visualizations can be integrated as easy as the first one. The tiling possibilities are instantly high- lighted while dragging (see Figure1top). Each view indicates five areas for dropping the visualization. Choosing the center replaces the current view with a new visualization, whereas taking one of the outer areas splits the view into two equally sized views (see Figure1below) and shows the new one in the chosen area.

As a coordinated multi-view approach the system provides linking within the time dimension across all views using a timestep marker which can be moved by by a simple finger swipe. Further- more, a coordinated time range selection defined by two fingers, see Figure2), and a coordinated subset selection within the data records dimension are also provided. Multiple records can be chosen by tapping on them individually (not depicted).

3.1. Segment and Zoom

Segment and Zoom is an interaction technique that is inherently woven into our tiled view management to switch between a given (over)view and a selected detail range within one view as well as composing a new setup of overview and detail (see Figure3) by

(3)

Figure 2:A coordinated range selection for highlighting a certain time region across all views is simply defined by touching the two ends of that range with one finger per hand (right view) at the same time. A defined range can be readjusted later for all views by mov- ing it along the time with one finger in one of the views (not depicted).

Figure 3:Composing a new detail view from a given overview can be invoked by dragging a previously defined temporal range (as in Figure2) with one finger towards the desired position (left). Drop- ping the range (right) divides the given space (here vertically) between the old view (or any other view) and the new one according to the splitting rules. A defined time range can be combined with the selection of one or multiple records resulting in a new detail containing only a temporal range of a subset of the records (not depicted).

simply defining a time range (see Figure2) at the beginning. Dou- ble tapping the selected time range invokes a magnified preview of it in the same view. It can either be used further as a regular view in another scenario or it can be doubled tapped again to return to the previous view (not depicted). The second possible route is to create a new detail sliced out of the given (over)view by simply dragging the selection towards a suiting space where it can be dropped (according to tiling rules). After releasing, a new detailed view appears (see Figure3). Such aSegment and Zoomsetup can be recursively composed (by using the new detail as new overview) or created manifold by selecting and dragging other time periods, which may eventually result in a cascade of derived views.Segment and Zoomalso considers the record dimension orthogonal to the time dimension by intersecting selected time ranges with selected records. This yields a new detailed view containing this particular subset regarding the chosen time range and selected records/time series.

Figure 4: Two views show the same data with different vertical space available. Compressing a view reduces (bottom view) the vertical space for each Horizon Graph, which is compensated by an increasing number of foldings (and thus the number of saturation steps per color hue). Such a view can be compressed down to a minimal size (by so many foldings) that each of the Horizon Graphs results in a single line of pixels and thus is being transformed into a pixel-based visualization. Vice-versa enlarging a view (top) decreases the number of foldings in each Horizon Graph.

Figure 5:Two particular Horizon Graphs (stock prices of IBM and Microsoft) have been emphasized by Pinch and Fold meaning they have been almost fully unfolded except that the values above and below the initial threshold are still shown in the same space en- coded by red and blue.

3.2. Pinch and Fold

A view’s content has to be responsive to scaling according to the aforementioned interactions or when the view borders are directly dragged in order to increase or to decrease the space for a particular view. We propose a direct-touch based extension for scaling a set of Horizon Graphs [RP08], which integrates well with the Fo- cus&Context paradigm.

Each Horizon Graph is initially displayed with a certain number of foldings appropriate for the space vertically available. When the available space changes through an interaction, the number of foldings is automatically adapted. It means that, eventually the entire view can be interactively scaled down (by dragging its borders) to a minimal size (by so many foldings) so that each single horizon graph results in a single line of pixels. Therefore a set of Horizon Graphs is transformed into a pixel-based visualization by an increasing number of foldings even though this is not a completely smooth process due to the discrete folding levels. Figure4presents the same set of Horizon Graphs in two different views at two different sizes and therefore with different numbers of foldings. The lower one nearly achieves one Horizon Graph per pixel line. Al- though only a diverging colormap with different saturation levels

(4)

for the two colors (e.g. red and blue) remains for mapping the values, the visualization can still serve as an overview of the general trend and reveals potential outliers similar to classic pixel-based visualizations (see Figure4, bottom view). This adaptive folding approach can also be initiated within the view with individual Hori- zon Graphs in order to emphasize particular Horizon Graphs over the other ones. A vertical Pinch and Zoom gesture adaptively ad- justs the height of the Horizon Graph between the two fingers and simultaneously folds/unfolds it accordingly (see Figure5). Thus, it can be considered as a type of foldable multi-focal Focus&Context applied on a set of Horizon Graphs.

3.3. Temporal and United Focus

Deriving a new view from an existing range or invoking a tempo- rary preview has disadvantages either by occupying existing space or by switching an entire view twice. Focus&Context is intended to solve these problems by emphasizing a focus region while showing the rest in a compressed form. Our approach allows defining a temporal focus region easily by touching the visualization with two fingers of each hand. Dragging horizontally expands or shrinks the focus region to its desired size while adapting the space for the context accordingly. Creating multiple focus regions is possible (see Figure6), as well as creatingUnited Focusregions where a temporal focus region is combined with unfolded (via Pinch and Fold and, thus, emphasized) records resulting in a two-dimensional Focus&Context display as in Figure7.

Figure 6:Two temporal focus areas were defined along the time axis. The contextual information before, between, and after is shrunken in order to fit into the remaining space.

4. Implementation Details

The system is developed from scratch on Ubuntu Linux using a 55 inch 4k tabletop display equipped with a touch overlay of iac.technology. The touch input is handled via the TUIO protocol.

View management and tiling are entirely realized on top of a tai- lormade 2D scenegraph that is rendered directly on graphics hard- ware with OpenGL. The test data is stock market data gathered with daily granularity from stocks in major indices such as Dow Jones, SP500, Dax, and Tec-Dax. The system is not especially made for stock market data though. It works for most time-oriented data sets.

5. Conclusion and Future Work

This paper describes novel touch-centered paradigms for time- oriented data. Our touch vocabulary is woven into a view/screen tiling approach for creating and placing Overview and Detail view combinations instantly and recursively resulting in cascades of views. Since cascades of views can be derived from not only a single view but from many (and thus resulting in many cascades), future development has to focus not only on visually linking those

Figure 7:United Focus is a two-dimensional Focus&Context approach combining a temporal focus region (defined and resized by two fingers per hand) with two particularly emphasized records (Horizon Graphs) previously unfolded and expanded via Pinch and Fold (see Figure5). The other records remain in their vertical scale only being affected by the temporal focus.

views belonging to one cascade (maybe even on their individual parent-child relations within), but also on distinguishing these multiple cascades. Widening our use scenario towards multi-user support is going to make identifying multiple cascades of views even more challenging aside from other problems such as user identification and handling more than two hands on the display.

Our newPinch and Foldtechnique extends the construction, ap- pearance, and interactivity of Horizon Graphs. The obvious next step is enabling intelligent vertical scaling or folding of other visualization techniques. For pixel-based visualizations this could be realized, for instance, by either adaptive accumulation of data values while scaling or by a mipmap-type approach that directly serves the appropriate value for each pixel at each view size.

While our touch-based interface was motivated by the observation that the left and right boundary of a range can be directly and in parallel specified by two fingers or two hands, our visualization techniques could be operated by mouse-based or pen-based input as well. However, which of the interfaces is more efficient or preferred by users, remains to be studied in future work.

Acknowledgments

We thank the reviewers of this papers for their constructive suggestions, which helped to improve the paper and the implementation. This work was supported in part by the German Federal Ministry of Education and Research (BMBF) under grant 03IP704 (project Intelligentes Lernen), grant 03IPT704X (project Big Data Analytics), and grant 03PSIPT5A (project Provenance Analytics).

(5)

References

[AKK96] ANKERST M., KEIM D. A., KRIEGEL H.-P.: Circle Seg- ments: A Technique for Visually Exploring Large Multidimensional Data Sets. InVisualization ’96, Hot Topic Session, San Francisco, CA (1996).1

[AMST11] AIGNERW., MIKSCHS., SCHUMANNH., TOMINSKIC.:

Visualization of Time-Oriented Data, 1st ed. Springer Publishing Com- pany, Incorporated, 2011.1

[BLC12] BAUR D., LEE B., CARPENDALE S.: Touchwave: Kinetic multi-touch manipulation for hierarchical stacked graphs. InProceed- ings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces(New York, NY, USA, 2012), ITS ’12, ACM, pp. 255–264.

doi:10.1145/2396636.2396675.2

[BW08] BYRONL., WATTENBERGM.: Stacked graphs – geometry &

aesthetics. IEEE Transactions on Visualization and Computer Graphics 14, 6 (Nov. 2008), 1245–1252.2

[HHN00] HAVRES., HETZLERB., NOWELLL.: Themeriver: Visualiz- ing theme changes over time. InProceedings of the IEEE Symposium on Information Vizualization 2000(Washington, DC, USA, 2000), INFO- VIS ’00, IEEE Computer Society, pp. 115–.2

[HKA09] HEERJ., KONGN., AGRAWALAM.: Sizing the horizon: The effects of chart size and layering on the graphical perception of time series visualizations. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(New York, NY, USA, 2009), CHI ’09, ACM, pp. 1303–1312.2

[Kei00] KEIMD. A.: Designing pixel-oriented visualization techniques:

Theory and applications.IEEE Transactions on Visualization and Com- puter Graphics 6, 1 (Jan. 2000), 59–78.1

[KSS04] KEIMD. A., SCHNEIDEWINDJ., SIPSM.: Circleview: A new approach for visualizing time-related multidimensional data sets. InPro- ceedings of the Working Conference on Advanced Visual Interfaces(New York, NY, USA, 2004), AVI ’04, ACM, pp. 179–182.1

[KWK^∗10] KAMMER D., WOJDZIAK J., KECK M., GROH R., TARANKOS.: Towards a formalization of multi-touch gestures. InACM International Conference on Interactive Tabletops and Surfaces(2010), ACM, pp. 49–58.2

[LAB^∗09] LAMMARSCHT., AIGNERW., BERTONEA., GÃDRTNER^ˇ J., MAYRE., MIKSCHS., SMUCM.: Hierarchical temporal patterns and interactive aggregated views for pixel-based visualizations. In2009 13th International Conference Information Visualisation (July 2009), pp. 44–50.2

[PVF13] PERINC., VERNIER F., FEKETEJ.-D.: Interactive horizon graphs: Improving the compact visualization of multiple time series. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(New York, NY, USA, 2013), CHI ’13, ACM, pp. 3217–3226.2 [RP08] REIJNER H., (PANOPTICON SOFTWARE): The development of the horizon graph. Online, 2008. www.stonesc.com/Vis08_

Workshop/.../Reijner_submission.pdf.1,2,3

[RROF18] RIEHMANNP., REIBERTJ., OPOLKA J., FROEHLICH B.:

Touch the time. Online, 2018. https://uni-weimar.de/

projekte/vr-scratch/videos/TTT.mp4.1