The Effect of Directionality on Cognition in Right-to-Left Languages

Boris Gorelik, Nabeel Sulieman

Abstract

What we know
Much of graph perception relies on conventions that seem standard and intuitive. One such convention is that numbers on a graph increase from the bottom up and from left to right. This left-to-right directionality agrees with the left-to-right writing systems used by most of the world's population. However, around 8% of the world uses right-to-left (RTL) scripts such as Arabic and Hebrew. Previous research has shown that RTL people perceive graphical information differently from their left-to-right (LTR) counterparts.

Nevertheless, no empirical study tests how this perception differs when an RTL reader sees a graph. Here, we present a study performed with English, Hebrew, and Arabic speakers in which we measured the RTL effect on visual perception. We also provide some guidelines to mitigate the challenges that we discovered.

What we did
We constructed a series of visual stimuli to test the differences in the perception of LTR and RTL audiences. The interpretation of some of these stimuli does not depend on horizontal direction (e.g., which circle is larger). Other images require the observer to determine whether a sloped line increases or decreases. The participant needs to decide where on the graph the line begins to do so.

We constructed a website that allowed us to expose volunteers to the stimuli while asking them to select one interpretation out of three. We recorded the volunteers' demographic data, responses, and response times. We hypothesized that if RTL people are "confused" by their primary reading direction, we will see a larger discrepancy and slower response times than their LTR counterparts.

What we found
We collected responses from 45 English-speaking, 86 Hebrew-speaking, and 11 Arabic-speaking volunteers. We couldn't find any difference between LTR and RTL volunteers in direction-insensitive stimuli. However, RTL participants needed more time to answer questions regarding direction-sensitive images. Their responses were also less coherent than those of the LTR responders.

What we conclude
In this empirical study, we demonstrated and quantified the difference in perception of horizontal direction in populations whose primary reading direction is right to left. Our results suggest that when other factors are equal, RTL readers (about 8% of the global population) will need more time and effort to decide the direction of a horizontal graph. We suggest mitigating this phenomenon by adding multiple visual clues of the horizontal axis directions. These clues include adding an arrow to the axis and providing numbered axis ticks.

Keywords: RTL, data visualization, pre-attentive attributes, diversion

Introduction

Data visualization is important in technical and academic fields and in day-to-day interfaces such as machinery and computers. It surrounds us, from automobile dashboards to financial documents and news sites. It is also essential in studying and communicating data-intense topics. Due to limited attention spans and perceptive capacities, charts must be precise and easily understandable. Otherwise, the author risks being misunderstood or ignored.

It is widely accepted that significant parts of how we perceive visual information are due to visual processing that happens without our conscious knowledge, called "preattentive attributes" (for example, see Albustin, Andreas, et al. "Preattentive processing." Course material. Graz University of Technology (2010).].

While most of the world's population reads and writes text from left to right, a significant portion reads and writes from right to left (RTL). Since reading is a central part of information consumption, the difference in reading direction may lead to variations in visual perception. As we will show below, these differences are very prominent in how a person analyzes graphs with temporal information. This study aims to find empirical evidence of the effect of reading direction on the perception of temporal information in graphs.

We will begin our discussion with a short overview of data visualization and what is already known about the unique characteristics of people who use RTL writing systems. We will then outline our study, present our results and discuss their implications.

On the Importance of Data Visualizations

image1

Scholars have been using visual elements to summarize complex topics since the dawn of the scientific age. For example, figure 2 shows a graphical depiction of planetary observation data recorded as early as the tenth century BC. As time passed, attempts have been made to standardize data visualization and explore the theory behind it. Willian Playfair made one of the earliest attempts to formalize data visualization in his 1798 work Lineal Arithmetic [Playfair, William. Lineal Arithmetic Applied to Shew the Progress of the Commerce and Revenue of England during the Present Century ; Which Is Represented and Illustrated by Thirty-Three Copper-Plate Charts. ... By William Playfair .. Printed for the Author, and Sold by A. Paris, 1798.].

image2

Figure 2 Planetary movements shown as cyclic inclinations over time, by an unknown astronomer, appearing in a 10th century appendix to commentaries by A. T. Macrobius on Cicero’s In Somnium Scripionus. Source: Funkhouser (1936, p. 261).

When we consider the goal of data visualization, we may divide this field in two, partially overlapping regions: exploratory and explanatory visualization tasks. Exploratory data visualization aims to aid the researcher in understanding the problem they are trying to solve. Explanatory data visualization aims to help a person explain or convince the reader of a particular idea. Vanessa Echeverria and co-authors extensively discussed the difference between the two areas in a paper published in 2018 [REF: https://learning-analytics.info/index.php/JLA/article/view/6114].

There is a great body of research exploring how we, humans, pay attention to visual information and deal with informational overload. Kahneman and Triesman suggested [ref: https://www.sciencedirect.com/science/article/pii/S0042698996001113] the existence of preattentive information attributes. According to this theory, which was later corroborated several studies, preattentive attributes are loose collections of basic features that allow us to make decisions about the information we perceive in a shorter period than the minimum required to pay attention.

For example, when we look at Figure 3A, we notice that it is a black-and-white image composed mainly of circles. We also immediately see the bright circle in the middle of the figure. We do so since the contrasting bright color is a preattentive attribute of Figure 3A. Similarly, we immediately notice the bright triangle in Figure 3B. However, most readers will need some effort to spot the hexagon on that same picture.

image3

Figure 3: Examples of preattentive attributes. Note how the hexagon in figure 3B is hard to spot.

Graph composition doesn't only affect our ability to spot elements. It also affects what we think of the information presented in a picture. For example, most people will perceive the leftmost square in Figure 4 as brighter than the rightmost one, even though all the squares in this figure have the same shade of gray.

image4

Figure 4 Effect of contrast

The importance of preattentive attributes comes into play in a shortage of time. Given enough time, guidance, and determination, most people will be able to correctly identify all the relevant features in the examples mentioned above. However, in the age of attention economy and informational overload, graph creators are competing with distractions. This limit of attention span can lead to misunderstandings or misguiding interpretations of data.

Temporal data and the time direction

An essential form of data visualization is the type of visualization that depicts temporal data, which are concepts that depend on time or a time-like variable. Most commonly, these visualization types use the horizontal axis to represent the progress of time. The direction of time here becomes crucial for the correct interpretation of a graph. For example, the axes in Figure 5 lack numbers or arrows. Determining whether the depicted line increases or decreases requires first determining where the line starts.

image5

Figure 5 Is this line increasing or decreasing?

Written text is a form of graphical representation of information. This information has an explicit temporal component, as it defines the order in which it needs to be read and written. If the horizontal dimension represents a form of progress or time, the task is to decide which direction the time progresses.

The majority of human populations use left-to-right (LTR) writing systems. Therefore, it is not surprising that most temporal graphs choose the same direction for time. This choice isn’t that trivial for people who write and read from right to left.

Perception differences between RTL and LRT people

Some languages (such as Arabic, Farsi, and Hebrew) are written exclusively from right to left, while other languages (such as Hindustani, Swahili, and Kurdish) use both LTR and RTL writing systems, depending on usage circumstances. Overall, around 600,000,000 people, or 8% of the human population, live in countries that officially use a right-to-left language. Should graphs designed for the RTL audience encode the temporal dimension in the LTR or RTL direction? What about other graphical depictions of time and progress, such as logos and illustrations? Due to the preattentive processing, this decision may affect the accuracy and speed of consuming such information.

Since reading and writing are so central to everyday activity, researchers have been studying the differences in visual perception between people who read from left to right and those who read from right to left. For example, in a 1987 study, Vaid and Singh [ref https://oaktrust.library.tamu.edu/bitstream/handle/1969.1/158729/Vaid and Singh 1989 Asymmetries in the perception of facial affect.pdf?sequence=1&isAllowed=y] examined the perception of asymmetric composite faces and noted that LTR but not RTL readers demonstrated a significant preference for the left hemifield. In another study, Chokron and De Agostini [ref https://www.sciencedirect.com/science/article/abs/pii/S0926641000000215] presented Hebrew (RTL) and French (LTR) readers pairs of images and their mirror reflections. The authors noted a strong connection between the subject reading direction and the preferred image directionality. In a different study, Chokron and De Agostini presented 120 adults and children who read Hebrew and French with a line bisection task [ref https://www.sciencedirect.com/science/article/pii/0926641095000186]. Interestingly, this study demonstrated a significant difference in bisecting the line, even before formal reading learning.

Another experiment by Afsari et al. [ref https://jov.arvojournals.org/article.aspx?articleid=2552689] exposed participants to various visual stimuli and used eye-tracking equipment to measure the visual fixation point relative to the image center. The researchers examined two groups of participants: native Arabic, Urdu, or Farsi (RTL) speakers who are also fluent in German (LTR), and a group of native German speakers. According to that study, when LTR participants start examining an image, the average fixation point lies approximately 10% to the left of the image center. The initial fixation point of RTL participants lies 5% to the right of the image. It took about three seconds before the average fixation point of the two groups overlapped. In other words, the study has demonstrated that RTL and LTR readers look differently at graphs and images for as long as three seconds, which is a significant period of time in cognitive terms.

Current state of data visualization in the RTL environments

There is no standard approach toward the time direction in data visualization in RTL languages. The images below show several examples of graphs in Arabic and Hebrew. These graphs come from textbooks, formal reports, and websites. Note the lack of uniformity. The example in Figure 6 from the Jordanian Edraak platform is especially interesting. Note the logo at the upper right corner of the figure. This logo forms an arrow, which we assume represents progress and thus points “forward” and up. Since the primary language of this organization is Arabic, the arrow points from right to left. However, the progress that the graph tries to show happens from left to right, thus causing a contradiction.

image6

Figure 6 Ad example from Jordan, with X-axis representing years running from left to right, while the logo in the upper-right corner indicates progress in the RTL direction.

Figures 7 and 8 show examples of charts in Arabic and Hebrew respectively, where the horizontal X-axis shows time progressing from left to right. Figure 9, on the other hand, demonstrates an RTL language chart with time progressing from left to right.

image

Figure 7 Graph example in Arabic from Israel. The line presented here increases. The time on the X-axis runs from right to left.

image

Figure 8 A screenshot of a Hebrew graph that depicts the number of diagnosed and recovered COVID-19 patients in the Israeli city of Carmiel. Note how the X-axis is oriented from right to left.

image

Figure 9 Screenshot of a graph from a Hebrew website [link https://treasureteen.com/2017/08/13/%D7%A2%D7%9C-%D7%94%D7%93%D7%9E%D7%95%D7%92%D7%A8%D7%A4%D7%99%D7%94-%D7%A9%D7%9C-%D7%9E%D7%93%D7%99%D7%A0%D7%AA-%D7%99%D7%A9%D7%A8%D7%90%D7%9C-%D7%97%D7%9C%D7%A7-%D7%90/]. The time flies from left to right.

In several cases, the graphs are tightly coupled with the text they contain. For example, Figure 10 shows a Sankey diagram with Hebrew text. Sankey diagrams have a time-related variable encoded as an X-axis. Since the text within the diagram uses an RTL script, the direction choice of the graph in that example looks more natural.

image

Figure 10 Screenshot of a Sankey diagram with Hebrew text. The overall direction of this graph corresponds the overall direction of the text, namely RTL. Origin https://www.facebook.com/photo.php?fbid=10221374156721147&set=a.1294632283658&type=3

Methods

In this section we describe the experiment designed to measure differences in visual cognition in RTL languages.

1.5 Stimuli Types

For the purpose of our experiment we classify visual stimuli according to their sensitivity to spatial organization. Namely, we classify images as being direction-free (F) or direction-sensitive (S). Examples of such stimuli are presented in Figure 11.

image

Figure 11: Stimuli examples. Nine stimuli examples. The first two rows are direction-free (F) and the last row is direction-sensitive (S).

Direction-free (F) stimuli are ones whose interpretation does not depend on horizontal reading direction. For example, visually assessing the height (row 1 of Figure 11) or size (row 2 of Figure 11) of a shape does not depend on whether a viewer scans the image from right to left or left to right. Similarly, detecting a horizontal line (row 3, left of Figure 11) doesn’t depend on reading direction either. However, determining whether a non-horizontal line is ascending, or descending is not as simple and depends on what a person considers to be the line beginning. Note that the charts in Figure 11 don’t have any markings on their X-axis. This was done intentionally, forcing the viewer not to rely on additional information.

The vertical position stimuli contain two circles (row 1 of Figure 11). The left-side circle was filled (“black”) and right one was empty. The only difference was the relative vertical position of the circles. We asked the participants whether the black circle was above, below, or at the same height as the white one. Thus, this family of images can be divided into three categories: "up", "down", and "equal".

The relative size stimuli contain two circles (row 2 of Figure 11). As with the height stimuli type, we placed a filled circle at the left side of the image, an empty circle at the right side of the image, and we asked whether the black circle was bigger, larger, or of the same size as the white one. Thus, the size family of stimuli can also be categorized into three categories "up" (i.e. bigger), down (i.e. smaller), and "equal".

The last family of stimuli, the one where the participant is asked to determine the overall direction of a line, contained only a black line. Since we were only interested in the way the participants perceive the slope of a line, we wanted to remove any clues that will affect the answer. We omitted a vertical axis as we assumed that the location of the vertical axis will provide a clue of where the horizontal axis begins. If we adopt the left-to-right perspective, the lines in our stimuli (row 3 of Figure 11) were either increasing (up), decreasing (down) or kept their vertical position.

When generating the images, we made sure that the differences were obvious to the observer by setting minimal slope angles, as well as height and area ratios.

Study Outline and Stimulus Sequence

Figure 12 shows a schematic representation of the study outline. To perform the study, we constructed a web site that is capable of presenting a user with stimuli and questions framed in one of several languages. We used English, Arabic, and Hebrew, and jsPsych to present the questions and record user responses and the time needed to answer the question. The first screen a user sees is a three-language message that asks them to select the language they are most comfortable reading. After the user selects a language, all the subsequent screens use only the language the user has selected. We did so in order to prime the user with the language they have selected.

image

Figure 12 Study outline.

Next, we ask the participants to provide some basic information about themselves, such as the gender, age, country of residence, languages they know, etc. We did not collect any personally identifying information such as names or email addresses.

Next, we present the participant with two series of direction-free stimuli. First, we asked the users to evaluate the relative size of two circles. Next, we asked them to evaluate the relative vertical position of two equally sized circles. Each such phase consisted of nine stimuli in a random order. Finally, we show the participant a series of black lines, and ask them to decide whether the line is increasing, decreasing, or level. We expose the users to the three types of questions (area, vertical position, slope) to make sure they get used to the interface, and that the subsequent decisions are mostly influenced by the visual perception, and not by the interaction with our site. The information obtained during this experiment phase is not used for making any direct inferences. However, we use this information as a prior distribution in the subsequent modelling process, which is described in the following section. We refer to this phase as the build-up or calibration phase.

After completing the built-up phase, we show the user a random series of twelve stimuli that are a combination of all three stimuli types. We made sure each type of stimulus appears an equal amount of times in this sequence of images. We refer to this last experiment segment as the collection phase.

Every time we expose the participant to a stimulus, we record the time needed to provide an answer, the image which we showed to the person, the question, and the answer. If you would like to view this experiment in action, please visit test.direction-matters.com. Since this is a test site, please feel free to run through the experiment as many times as you like.

Metrics

Perception accuracy

We created each stimulus to have one of the three “directions” – up, equal, and down. Correspondingly, study participant can provide one of the three answers: up, equal, and down. We used the Bayesian approach to estimate response accuracy of the different study groups [REF to PyMC3 https://doi.org/10.7717/peerj-cs.55].

Dirichlet distribution (DD) [REF??? do we need a ref? https://en.wikipedia.org/wiki/Dirichlet_distribution] is a multivariate generalization of Beta distribution and is well suited to model choice probabilities. Our job is to estimate posterior distribution of P(R_r│D_d ), where R and D are the response and the true direction respectively; and r,d∈{up,equal,down} represent the response and the stimulus directions. Since each true direction is represented by a single DD vector, P(R_up│D_d )+P(R_equal│D_d )+P(R_down│D_d )=1 for each d.

The first step in Bayesian estimation is to select the prior distributions. We started with the naïve uniform distribution (i.e. all the parameters in the Dirichlet distribution equal to 1.0), where each choice is equally likely. Next, we collected the responses to direction-insensitive stimuli that we collected from all the participants during the calibration phase and used it to update the DD parameters. We used the updated parametrized DD as the prior distribution of our subsequent computations.

Finally, we used the information that we collected during the collection phase to estimate the distribution of P(R_r│D_d ) in the different populations that participated in the study, namely the participants who selected English, Hebrew or Arabic languages. We further split our analysis into two orthogonal groups: one for the direction-sensitive stimuli, and one for direction-insensitive ones. We use the resulting distributions to make inference on how likely the participants who are used to different reading direction to get confused while estimating whether a diagonal line represents an increasing or decreasing trend.

Response Time

In addition to collecting answers to our questions, we also recorded the time need by the participants to provide the response. Our initial intention was to check whether the RTL participants need more time to analyzed direction-sensitive stimuli. However, the participants filled in our questionnaires in their own time, in different places, and on different devices, including tablet computers and mobile phones. Thus, we found the analysis of the response time unfeasible. Nevertheless, we provide the response times as auxiliary data to this document, and we call the scientific community to study this aspect in a more controlled environment.

Results

Overall, 142 volunteers completed our survey. Responder distribution, as well as some basic demographic metrics, are summarized in Table 1.

Group N ♂/♀/Other[%] Age [Years ± Std] Age Distribution
English 45 51.1/46.7/2.2 34.2±13.9 image
Hebrew 86 30.2/58.1/11.6 37.1±18.1 image
Arabic 11 54.5/27.3/18.2 22.2±12.9 image
All 142 38.7/52.1/9.2 35.1±17.0 image

Table 1. Participant summary. Dot color in the Age Distribution column denotes responder’s gender: blue for male, red for female and black for other or unreported gender.

Response accuracy

We show the probability distribution of the response confusion matrix on Figure 13 (for direction-free stimuli) and 14 (for direction-sensitive stimuli).

We designed our stimuli in such a way that any stimulus that is not supposed to belong to the “equal” category is prominently seen as such. We even added a proper clarification to the explanation screens. Therefore, we assume that marking an “equal” stimuli as either “up” or “down” probably results from a random mistake. Similarly, marking an “up” or “down” stimulus as “equal” also corresponds to a random error. Therefore, we designate the corresponding cells in the confusion matrix with a gray background. Response cells that are most relevant to our study are the “up-up” and “down-down” for correct responses; and “up-down” and “down-up” for errors that might stem from a systematic difference in direction perception. Thus, we mark the correct cells with a blue background and systematic errors with a red one.

image

Figure 13. Similar probabilities of correct answers in direction-insensitive stimuli. Odds ratios (log scale) for correct stimulus classification. Each row represents the actual stimulus direction. Each column represents participants' response. The dots mark the maximum posterior estimation; the horizontal lines show the interquartile range of the rate estimation.

image

Figure 14. Less accurate responses by Hebrew and Arabic participants. Odds ratios (log scale) for correct stimulus classification. Each row represents the actual stimulus direction. Each column represents participants' response. The dots mark the maximum posterior estimation; the horizontal lines show the interquartile range of the rate estimation. Note the differences between the English vs. Hebrew and Arabic estimates in the upper left and lower right cells.

The graph matrices in Figures 13 and 14 represent "confusion matrices". To emphasize the effect of the reading direction on graph perception, we combined the responses located in the "mistake" corners of the confusion matrices (i.e upper left and lower right cells). These corners represent "genuine mistake", where a participant completely missclassifies the stimulus presented to them. If we combine the responses in these cells, group the Hebrew and Arabic responders in a single category, we can clearly see that the RTL participants have much higher chances to misunderstand the direction of a sloped line (Figure 15).

image

Figure 15. RTL people make more mistakeswhen reading direction matters. Error odds ratio (log scale) for answers that rely on reading direction. In this graph, we combined the Hebrew and Arabic participants to a single group named "RTL".

Works Cited

Playfair, William. Lineal Arithmetic Applied to Shew the Progress of the Commerce and Revenue of England during the Present Century; Which Is Represented and Illustrated by Thirty-Three Copper-Plate Charts. ... By William Playfair .. Printed for the Author, and Sold by A. Paris, 1798.

Acknowledgements

References