Does Navigation Always Predict Performance ? Effects of Navigation on Digital Reading are Moderated by Comprehension Skills

This study investigated interactive effects of navigation and offline comprehension skill on digital reading performance. As indicators of navigation, relevant page selection and irrelevant page selection were considered. In 533 Spanish high school students aged 11-17 positive effects of offline comprehension skill and relevant page selection on digital reading performance were found, while irrelevant page selection had a negative effect. In addition, an interaction between relevant page selection and offline comprehension skill was found. While the effect of relevant page selection was strong in good offline comprehenders, it was significantly reduced in weak offline comprehenders. The effect of offline comprehension skill was strong in students showing high rates of relevant page selection, while it was weak and insignificant in students showing low rates of relevant page selection.


Résumé de l'article
This study investigated interactive effects of navigation and offline comprehension skill on digital reading performance. As indicators of navigation relevant page selection and irrelevant page selection were considered. In 533 Spanish high school students aged 11-17 positive effects of offline comprehension skill and relevant page selection on digital reading performance were found, while irrelevant page selection had a negative effect. In addition, an interaction between relevant page selection and offline comprehension skill was found. While the effect of relevant page selection was strong in good offline comprehenders, it was significantly reduced in weak offline comprehenders. The effect of offline comprehension skill was strong in students showing high rates of relevant page selection, while it was weak and insignificant in students showing low rates of relevant page selection.

Introduction
In learning scenarios where materials are delivered online, e.g. in distance education programmes, students frequently need to progress on their task by navigating through a network of linked materials (or nodes). Specifically, during navigation they have to assess the relevance of the available hyperlinks, and decide which one may contain information useful to answer the question. If students access a node with relevant information, they may still have to assess if the information obtained from this node is enough, or if they need to access further relevant links to provide a complete answer. Think of a person who considers taking up studies in, e.g., Psychology in a distance learning setting, and accesses the website of, e.g., the Open University. Even before enrolling in a study program, they will need to access a number of hyperlinks, and read and integrate the documents opened by these, to find out about course contents, fees, terms of application, etc. After having enrolled, they will need to complete study assignments sent to them online. Again, completing these assignments, e.g., when searching for literature to complete an essay, they will need to engage with navigating documents online by accessing certain hyperlinks, and discarding others when they are not relevant to the task at hand. Thus, theoretically, navigation is a crucial process in online and distance learning. In addition, traditional or offline comprehension skills are needed to process the documents accessed through the navigation process (e.g., Salmerón & García, 2011). This means that in online learning scenarios as well, students need to decode words, parse the syntax of sentences, and execute local and global coherence processes to finally understand a document's contents (e.g., Kintsch, 1998). This article aims at a contribution to clarifying how navigation processes and offline comprehension skills interact in online learning scenarios. In the following section, we first review the existing literature on this interaction.

Processes
There is clear evidence that students' navigation, as indicated by their traces in log files, play a major role in online question-answering tasks (Organization of Economic Co-Operation and Development [OECD], 2011). Specifically, a large scale study involving the adolescents participating in the OECD Programme for International Student Assessment (PISA) 1 2009 electronic reading assessment revealed that students who displayed a more task-oriented navigation behaviour, as indicated by more visits to task-relevant pages, correctly responded to a higher number of questions. This result is in line with several other small scale studies that have looked at how adolescents and undergraduate students navigate in different learning tasks. These tasks include searching for information (Cress & Knabel, 2003;Hsu & Schwen, 2003), studying for a semester-long course (Puntambekar & Goldstein, 2007), reading for comprehension (Salmerón, Cañas, Kintsch, & Fajardo, 2005;Salmerón, Kintsch, & Cañas, 2006), reading to 1 PISA is an international assessment of student skills that is carried out every three years since the year 2000 in all OECD and a number of partner countries and economies. The core assessment is in mathematics, science, and reading in students aged 15, which in most countries means the end of compulsory education. See OECD (2010OECD ( , 2011 for details on PISA 2009. prepare a summary (Richter, Naumann, & Noller, 2003;Naumann, Richter, Flender, Christmann, & Groeben, 2007), or performing a science inquiry task (Goldman, Braasch, Wiley, Graesser, & Brodowinksa, 2012). In addition, OECD (2011) showed that students with good offline comprehension skills, as measured by a paper and pencil reading comprehension test, employed better navigation paths, which in turn improved even more their online performance.
While this mediating role of navigation on the relationship between offline comprehension skills and online performance has been already reported in small scale studies (Naumann, Richter, Christmann, & Groeben, 2008;Salmerón & García, 2011), less is known about the potential moderator effect of offline comprehension skills on the relationship between navigation and performance (e.g., navigation may predict online performance only for students with sufficient offline comprehension skills). The goal of this study is to shed light on this aspect.
The fact that navigation, as measured by log files, predicts students' performance in several online learning tasks has raised the attention of scholars due to its important implications on a theoretical and on a practical level (e.g. Güyer, Atasoy & Somyürek, 2015). On a theoretical level, navigation is important because it is highly interlinked with self-regulation learning (SRL) in online scenarios. SRL models describe students' decisions in complex learning scenarios, such as deciding which hypertext sections to study and which one not. These decisions are based on students' reflections on the constraints imposed by the scenario, their task goals, and the revision of their performance during the task (Winne & Hadwin, 1998). In addition, SRL models assume that learners try to optimize both achievement and how achievement is reached (Winne & Baker, 2013). In online learning, readers are provided with tasks that require a self-directed selection and organization of text materials. In terms of achievement, there is ample evidence that increased self-regulation in online learning goes together with better learning outcomes (e.g. Azevedo & Cromley, 2004;Azevedo, Greene, & Moos, 2007;Azevedo, Moos, Greene, Winters, & Cromley, 2008). In terms of how achievement is reached, there is evidence that better selfregulation goes together with better navigation, and that better navigation partly mediates effects of self-regulation on learning outcomes in online learning scenarios (Richter, Naumann, Brunner & Christmann, 2005;Salmerón, Kintsch, & Kintsch, 2010). Thus, altogether, the analysis of navigation behavior can provide rich information to self-regulated models of learning (Winne, 2010).
On a practical level, the fact that navigation, as measured by log files, predicts students' performance in several online learning tasks makes it useful for intelligent online systems. Log files can be traced to build user models, and recommendations can be provided during the learning session on the basis of students' navigation. For example, if the system detects that a student is navigating task-irrelevant nodes, it may react, prompting her to focus on more relevant ones (e.g. Puntambekar & Stylianou, 2005;Vidal-Abarca et al., 2014;Winne, & Hadwin, 2013).
The current study questions the widespread assumption that navigation univocally predicts performance in question-answering online tasks. Specifically, we tested the assumption that the positive relationship between navigation and performance may be moderated by students' offline comprehension skills. To the best of our knowledge, the potential moderator effect of comprehension skill on the relation between navigation and performance has not yet been investigated. Theoretically, moderation takes place when offline comprehension skills and navigation interact statistically in predicting online reading performance. A student might lack offline comprehension skills, but still have the skill necessary to navigate a digital text. For example students may match words in the question to words in the text, without a thorough examination of the relevance of that particular text section (Cerdán, Gilabert, & Vidal-Abarca, 2011;Salmerón, Cerdán, & Naumann, 2015), or they may use typographical cues of hyperlinks to guide their navigation (Rouet et al., 2011). In this case, however, despite an apparently good navigation, comprehension outcomes will be poor. This is because a text that is 'created' through a readers' self-guided selection and ordering of text materials, i.e. navigation, has to be read and understood eventually. Thus, according to this reasoning the effect of navigation on digital reading performance will be conditional on offline comprehension skills. In a student with good offline comprehension skills, a navigational path containing a large percentage of task-relevant materials will lead to good comprehension outcomes. A student with poor offline comprehension skills, in contrast, will not be able to utilize such a text.
Our study aimed at testing the hypothesis that the relation between navigation and performance is moderated by students' offline comprehension skills. To do so we used a large ( N=533) agestratified sample comprising students aged eleven to 17 (see Table 1).

Method
Subjects 533 Spanish students participated in the study. Students were sampled from six different schools in the province of Valencia, Spain. Within each school, one complete classroom of 7 th , 8 th , 9 th , and 10 th graders was sampled. Schools and children were recruited to participate under specific agreement of collaboration between principals, regional educational authorities and the research team. Navigation. Log files were used to analyze students' navigation behavior. We used as an index of the navigation quality the percentage of task-relevant pages visited for each item (relevant page selection). The number of task-relevant pages visited reflects the process of taskaccurate selection of learning or reading materials from online sources and can thus be assumed one crucial aspect of the navigation process, and has thus been shown to be associated with the quality of comprehension and learning (e.g. Cress & Knabel, 2003;Goldman et al., 2012;Naumann et al., 2008). Pages were considered relevant for a particular question if students needed to access them in order to locate information useful to answer that question. Thus, this index not only considers if a participant accessed or not the page or pages that contained useful information, but more precisely the extent to which she followed the path leading to those useful pages. To rule out the possibility that this index merely reflects the length of a navigational path, we considered in addition the percentage of irrelevant pages that were accessed by a student in each task (irrelevant page selection). While we predicted a positive effect of relevant page selection on digital reading performance, we predicted a negative effect of irrelevant page selection.

Procedure
Students were tested in their classrooms during school hours in two one-hour sessions. In a first session, students completed the offline comprehension skills test. After completion, they filled in two questionnaires asking for demographic information and about the frequency of their use of computers. In a second session, students completed the digital reading performance measure in the schools' computer lab. Data was collected using USB devices.

Statistical Analysis
To test our hypothesis about the moderator role of offline comprehension skills on the relation between navigation and performance, we estimated a linear mixed model, using the R environment (R Development Core Team, 2012) with the package lme4 (Bates, Maechler, & Bolker, 2012). The predictor variables offline comprehension skill, relevant page selection, and irrelevant page selection were entered as fixed effects. In addition, the interactions of offline comprehension skill with relevant page selection and irrelevant page selection were entered as fixed effects. Since the predictor variable offline comprehension skill could be expected to covary with age, and thus be confounded with other age-related variables, we entered students' age as a control variable. All predictor and control variables were z-standardized. Since complete classrooms were sampled, we entered a random classroom intercept to account for dependencies between students coming from the same classroom. As a measure of effect size, we report as a default the proportion of variance uniquely accounted for by each fixed effect (marginal Δ R 2 following Nakagawa and Schielzeth, 2013). The significance of the random classroom intercept effect was evaluated using a simulation-based exact likelihood test with the package RLRsim (Scheipl, Greven, & Kuechenhoff, 2008).

Results
Descriptive statistics and correlations for all variables in the model are given in Table 1 (rows 5-9).
Recall that our hypothesis claims that the effect of relevant page selection on online comprehension is enhanced by offline comprehension skills, since without those appropriate skills, students displaying whatever navigation behavior will not be able to completely understand a digital text. This means that in a regression model with offline comprehension skills and relevant page selection as predictors, plus an interaction term, a positive regression coefficient should emerge for the interaction, indicating an increasing impact of relevant page selection on digital reading performance with increasing offline comprehension skills. In contrast, no such interaction effect was expected for irrelevant page selection. Selection of irrelevant pages can be seen as detrimental to digital reading performance, regardless of a student's offline comprehension skill. Model results are summarized in Table 2. * p < .05, ** p < .01, *** p < .001.
The random effect for classroom was significant, meaning that there was systematic betweenclassroom variance in digital reading performance over and above what was explained by the fixed effects in the model (see Table 2).
Analysis of fixed effects revealed a large positive effect of relevant page selection (Table 2, row 3), while irrelevant page selection had a negative effect as expected (Table 2, row 4), which was however small. Also as expected, offline comprehension skill had a medium-sized positive effect (Table 2, row 5). In addition to these main effects, there was a positive effect for the interaction between relevant page selection and offline comprehension skill (Table 2, row 6). The interaction between offline comprehension skills and irrelevant page selection was not significant (Table 2, row 7). The control variable age had no effect over and above the other predictors in the model (Table 2, row 8).
In line with our hypothesis, the positive sign of the regression coefficient for the interaction between relevant page selection and offline comprehension skills indicated that positive effects of relevant page selection on digital reading performance were accelerated with increasing offline comprehension skills. Accordingly, positive effects of offline comprehension skills were accelerated with increasing rates of relevant page selection. To further interpret this interaction, we conducted simple slopes analyses (see Aiken & West, 1991).
Firstly, we estimated the effects of relevant page selection on digital reading performance conditionally on high offline comprehension skills (one standard deviation above the mean, see the straight line in Figure 1a) and low offline comprehension skills (one standard deviation below the mean, see the dotted line in Figure 1a).
good offline comprehension skills are not sufficient in themselves to produce good digital reading performance. Rather, if students fail to comply with demands of relevant page selection, the otherwise strong and positive association of offline comprehension skill and digital reading performance is no longer significant.
One possible explanation for these results is a threshold model. According to such a model, when a critical threshold of offline comprehension skills is reached, students can benefit from their efficient navigation to improve their digital reading performance. It is a reasonable assumption that students would need to score on the PISA reading proficiency level 3 (out of 6) or above. This means, for instance, that they are able to, e.g., find multiple pieces of information in a text, or link information from different texts to one another (see OECD, 2010, for a detailed description). This assumption however needs to be corroborated by future research as the offline comprehension skills measure used in this study is not linked to the PISA reading scale.
It is strong comprehenders who benefit from efficient navigation of task relevant pages. Why do, in contrast, poor readers not benefit so much from their navigation? From the perspective of a developmental model, one can argue that these students still fail on matching their navigation goals to the available links by means of inferences. Thus, their assessments of a link's relevance might rely on more superficial processes, such as links' typography (Rouet et al., 2011) or word matching (Cerdán et al., 2011;Salmerón et al., 2015), e.g., if the goal and the hyperlink share a particular word. By using those cues to navigate, students may still access pages that are relevant for their goal, but nonetheless they won't be able to make complete sense of the information on those pages. Log files provide information about the percentage of relevant pages visited by students, but they don't give a complete picture of which cognitive operations students actually use to select those pages. A possible way to clarify this issue is to combine the information from log files with other online measures, such as students ' verbal protocols (e.g. van Gog, Paas, van Marriënboer, & Witte, 2005;Richter et al., 2005). Future research should clarify this issue.
In claiming a moderation effect of offline comprehension skills for the relationship between navigation and online performance, some precautionary remarks are in place as well. First, while it was one crucial property of navigation that was investigated in the study reported here, the selection of relevant (and irrelevant) text materials, this is by no means the only important property of a navigational path. For instance, coherence of the navigational path has been established as another variable that is predictive of comprehension and learning outcomes in reading and comprehending digital text (Amadieu & Tricot, 2008;Naumann et al., 2007;Richter et al., 2005;Salmerón et al., 2005;Salmerón, Kintsch, & Cañas, 2006), as has the use of navigation help devices such as topical overviews (Dias & Sousa, 1997;Salmerón & García, 2011;Su & Klein, 2006). For these properties of navigation, for the time being, we do not know whether, and how, their impact on performance is moderated by students' skills.
Second, although in our study offline comprehension skills moderate the relationship between navigation and performance, navigation has a positive effect in most students. Other patterns of moderation can be observed. For example, a particular navigation pattern may be useful for students with or without a particular skill, but may have no effect on others. Although it is not directly linked to navigation, related evidence on the effect of overviews in hypertext exemplify these potential moderator patterns. In this line, Salmerón, Baccino, Cañas, Madrid, & Fajardo (2009, study 1) found that providing a thematic overview for a hypertext had a positive effect on students' comprehension, only if the hypertext was difficult and if students had low background knowledge on the topic. Finally, while offline comprehension skills is a strong predictor of navigation, research has identified other students' individual variables that are also important to articulate an efficient navigation, and thus their potential role as moderators should be studied in the future. One such important variable is background topic knowledge (e.g., Coiro, 2011). We could expect that, given a similar percentage of relevant pages visited, students with high background knowledge may fill in the existing gaps in their response, while students with low background knowledge may have difficulties overcoming an incomplete response generated through their navigation (cf., Taub, Azevedo, Bouchet, & Khosravifar, 2014).

Implications for Research on Navigation
What are the potential implications for the moderated relationship between navigation and online performance? Here, the present research carries two messages. First, at a theoretical level, our results imply that similar navigation traces may be reflecting different cognitive and metacognitive processes. This is a challenge for using navigation traces to inform and enrich selfregulation models of learning. A way to limit the number of potential processes underlying the selection of a particular navigation path is to include theoretically motivated hyperlinks, the selection of which can be easily linked to a unique theoretical explanation (Winne, 2010). For example, a recent question-answering study using Wikipedia (Salmerón et al., 2015) included three types of hyperlinks embedded in relevant sections of the hypertext: a) relevant uncued links that paraphrased the main idea of the question, and that led to a hypertext node with useful information to answer the question; b) irrelevant cued links that included a word that matched with a word in the question, although it was not representative of the main idea of the question, and the link led to an irrelevant node, and c) irrelevant uncued links that didn't include a paraphrase or a word match to the question. Results from the navigation patterns followed by the students allowed to identify which relevance cues (semantic and/or word matching) they were using to navigate the hypertext. Second, at a practical level, the existence of a moderation effect indicates that navigation alone, at least the navigation index used in this study, i.e., percentage of relevant pages accessed, can't be used to accurately predict students' performance in online question-answering tasks. An intelligent system that reacts to students' navigation behavior, without considering important student variables, such as offline comprehension skills, will generate an inaccurate representation of the user, which may lead to less useful recommendations. For example, if a system identifies that a user is deviating from the optimal path to find relevant information, recommending an optimal route may be useful if the student has the necessary skills to realize that she is in an unfruitful path, but it may be less efficient if the student does not have the necessary reading skills to do so. Thus, in designing distance learning materials clearly usability in terms of navigation is important, i.e., supporting students to actually identify the relevant parts of the materials given the aims of study. What might however be equally important as suggested by the present results is to look at the text's readability and difficulty itself. Since online learners' offline comprehension skills might be measured before students engage with the materials, they cannot be changed in the short term. Thus, facing students with comprehension difficulties, for a course designer, it will be of importance to create materials as easy to comprehend as possible.
Otherwise, as suggested by the present results, increasing the materials' usability in terms of navigation won't help much.
As we see it, the present analyses however already make a strong argument that while navigation does play a crucial role in digital reading performance, its effect is moderated by the more traditional offline comprehension skills. This also means that instructional programs which aim at strengthening students' navigation processes, preparing them for a digital 21 st century world where open and distance learning scenarios are ubiquitous, should take students' comprehension skills into account. Probably, students who struggle with comprehension will need support to strengthen their comprehension skills first, before they can benefit from instruction targeted at their preparedness to adequately select hyperlinks in online reading scenarios.

Limitations
Finally, some cautionary remarks about limitations of the present results are in place. First of all, our data base was correlational. This means that the present results should be backed, e.g., by training studies, where in the best case offline comprehension skill and navigation processes are manipulated experimentally. Second, while we analyzed data from high school students in this research, open and distance learning is frequently, and in the future will be, aimed at adults (e.g., Muñoz, Redecker, Vourikari, & Punie, 2014). While there appears to be some evidence that in adults comprehension skills interact with navigation support (Naumann et al., 2007), little is known though about possible interactions between comprehension skill in navigation behavior in adult distance learners. Cleary, future research will have to shed light on this issue.