Analytics to Literacies: The Development of a Learning Analytics Framework for Multiliteracies Assessment

The rapid advances in information and communication technologies, coupled with increased access to information and the formation of global communities, have resulted in interest among researchers and academics to revise educational practice to move beyond traditional ‘literacy’ skills towards an enhanced set of “multiliteracies” or “new media literacies”. Measuring the literacy of a population, in the light of its linkage to individual and community wealth and wellbeing, is essential to determining the impact of compulsory education. The opportunity now is to develop tools to assess individual and societal attainment of these new literacies. Drawing on the work of Jenkins and colleagues (2006) and notions of a participatory culture, this paper proposes a conceptual framework for how learning analytics can assist in measuring individual achievement of multiliteracies and how this evaluative process can be scaled to provide an institutional perspective of the educational progress in fostering these fundamental skills.


Article abstract
The rapid advances in information and communication technologies, coupled with increased access to information and the formation of global communities, have resulted in interest among researchers and academics to revise educational practice to move beyond traditional 'literacy' skills towards an enhanced set of "multiliteracies" or "new media literacies". Measuring the literacy of a population, in the light of its linkage to individual and community wealth and wellbeing, is essential to determining the impact of compulsory education. The opportunity now is to develop tools to assess individual and societal attainment of these new literacies. Drawing on the work of Jenkins and colleagues (2006) and notions of a participatory culture, this paper proposes a conceptual framework for how learning analytics can assist in measuring individual achievement of multiliteracies and how this evaluative process can be scaled to provide an institutional perspective of the educational progress in fostering these fundamental skills.

Introduction
The development of a literate population is one of the most recognizable goals of public education. Central to this goal has been defining what counts as literacy for contemporary society. In the more traditional sense, literacy was, and continues to be, deeply enmeshed with the written word (Kress, 2003). As Jim Dator (2005) argues, "seldom has a technology been the subject of more worship than the word is in literate cultures" (p. 202). This emphasis on the written word stems from the perceived relationship between basic literacy skills and future economic prospects. Even in the industrial age, the cultivation of a literate population (reading, writing and basic arithmetic) was viewed as an essential ingredient for achieving democracy, economic growth, and social stability (Kalman, 2008). In essence, an individual's future social and economic prosperity was related to their capacity to read and write (Leadbeater & Wong, 2009). It is little wonder then that literacies have continued to be so intimately linked to economic benefits, such as higher socio-economic status, increased job opportunities, and increased wealth within the community (UNESCO Education Sector, 2004).
Basic literacies are as relevant today in the information age as they were previously, especially given that the Internet and mobile devices remain as heavily text-based technologies (Greenhow & Robelia, 2009;Warschauer, 2007). However, the pervasiveness and pace of change associated with new forms of media and the contexts in which they are applied places additional expectations on what it now means to be "literate" (Anstey & Bull, 2006;Huijser, 2006). That is, the expectation now exists that individuals have at their disposal a diverse set of skills and cultural competencies necessary to navigate the various forms of digital communication and participation in a global society. While not understating the continued importance for reading, writing, and numeracy skills, an emphasis placed solely on those 'basics' ignores the broader changes that have occurred in the cultural, social, and economic landscape and the speed of access to technologies and information that are now prevalent in today's society (Kalantzis, Cope, & Harvey, 2003). A society that is increasingly reliant on technology for information access and communication alongside the globalization of information requires its citizens to effectively utilize a greatly expanded skill set to encompass cultural and new media competencies as well as a recognition of the various contexts in which they are applied (Coiro, Knobel, Lankshear, & Leu, 2014).
If an expanded form of literacy, multiliteracies, is the key to individual and community wealth and wellbeing within a society, then it is crucial to the value of education to establish measures regarding how well students in the education system are mastering these fundamentals. Not only is there a need to assess individual students on their progress and mastery of skills in multiliteracy, but it is important to measure systemic progress and attainment of multiliteracies of the education system and society as a whole. Measurement of the literacy of a population, in the light of its linkage to individual and community wealth and wellbeing, is an essential and fundamental measure of the success of formal education.
This paper begins with an examination of multiliteracies in order to create a defined set of skills that can be considered essential literacies for today's society. Following this review, the potential assessment and learning activity artifacts that might be generated in the process of learning multiliteracies are discussed. The trend for increasing adoption of online and blended modes of learning continues unabated at a global level and as such there are now unprecedented levels of trace data that can be harnessed to inform teaching and learning practice. Given the potential artifacts that learners generate in these online learning environments, this paper proposes a conceptual framework for using learning analytics to measure the development of multiliteracies across a group of learners. This paper outlines a framework for moving analytics from what has been previously described as the "low hanging fruit" 1 up the ladder to richer and more complex multi-dimensional analytics in education. The conceptual framework proposed in this paper emphasizes the identification of key multiliteracies based on a review of frameworks created by media scholars and then suggests possible data sources and analytics strategies that will enable a means of measuring and determining the impact of those literacies. Validation of the learning analytics/multiliteracy framework is beyond the scope of this paper but it is expected that detailing the relationship between literacies, artefacts, and analytics techniques will lead to empirical work to test, validate, and revise the frameworks offered by media scholars.

Multiliteracies
Recognition of the dramatically changing nature of what it means to be literate in the so-called 'information age' has seen the rise of discussions within educational research around the importance of students developing "multiliteracy" skills (I. Brown, Lockyer, & Caputi, 2010;Cope & Kalantzis, 2000a;Haythornthwaite & Andrews, 2011). As first described by the New London Group (1996), the term multiliteracies extends the scope of traditional literacy to include the diversity of media and modes of communication that are now available to learners and the varying contexts in which they are utilized. In discussing the genesis of the term multiliteracies, Gee (2009) noted that literacy "needed to be viewed as embedded in multiple socially and culturally constructed practices, not seen as a uniform set of mental abilities or processes" (p. 196). In essence, the New London Group (NLG) challenged the existing singular view of the term literacy recognizing the multiplicity of communications available to learners both as producers and consumers and the increasing cultural and linguistic diversity that is prevalent today (Cope & Kalantzis, 2000b Since establishing "a pedagogy of multiliteracies" (Cope & Kalantzis, 2000b;New London Group, 1996), much conceptual and theoretical work has been undertaken in the literacy field (e.g., Gee, 2007;Kress, 2003;Lankshear & Knobel, 2003;Muspratt, Luke, & Freebody, 1998). The goal of this paper is to bring a new perspective of how to assess and evaluate the development of new literacies and the pedagogical activities that underpin such characteristics and skills, rather than to establish an authoritative definition or review of literacies.

A Participatory Culture
While further theorizing of the changing nature of and intersection between literacy, learning, and digitality needs to continue, there is acceptance that multiliteracies involve an increasing set of social skills that draw upon an ever expanding set of technologies, media, and discourses (Gee, 2007;Unsworth, 2001). In this context, viewing literacy through a sociocultural lens encompasses the fundamentals of literacy (e.g., reading, writing and meaning making) within embedded social practice (Lankshear & Knobel, 2007). Henry Jenkins (2006) emphasizes this point in noting that "the new literacies almost all involve social skills developed through collaboration and networking" (p. 4).
The complexity of the information, media, and technology environment that learners draw on in their day to day academic and social activities, as well as the types of skills necessary to be productive community members, is captured in Jenkins, et al.'s (2006) notion of a participatory culture. Jenkins and colleagues describe a participatory culture as one with "relatively low barriers to artistic expression and civic engagement, strong support for creating and sharing creations, and some type of informal mentorship whereby experienced participants pass along knowledge to novices" (p. xi). Educational engagement in this form of practice is strongly aligned with more Vygotskian-influenced approaches to learning such as social learning (J. S. Brown & Adler, 2008), game-based learning (Gee, 2007), self-directed learning (Garrison, 1997), and communities of practice (Lave & Wenger, 1991 (Jenkins et al., 2006) or "cultural competencies and social skills" (p. xiii) necessary to be productive participatory citizens. The competencies and skills necessary for a participatory culture include networking, collaboration, creativity, citizenship, and communication within a multiplicity of modes and mediums. Table 1 outlines Jenkins et al.'s (2006) new media literacies stressing the transition of literacy practice from individual to community. Simply put, for 21 st century education these skills and attributes are considered to be the 'new basics'. As such, the plurality of these new literacies also calls for new and diverse forms of assessment to be developed and implemented within the formal education context. Table 1 Overview of New Media Literacies (Jenkins et al., 2006) New media literacies Description Play Capacity to experiment with one's surroundings as a form of problem-solving. Performance Ability to adopt alternative identities for improvisation and discovery.

Simulation
Ability to interpret and construct dynamic models of realworld processes. Appropriation Ability to meaningfully sample and remix media content Multitasking Ability to scan one's environment and shift focus as needed to salient details.

Distributed cognition
Ability to interact meaningfully with tools that expand mental capacities.

Collective intelligence
Ability to pool knowledge & compare notes with others toward a common goal. Judgment Ability to evaluate the reliability & credibility of different information sources. Transmedia Navigation Ability to follow the flow of stories and information across multiple modalities. Networking Ability to search for, synthesize, and disseminate information. Negotiation Ability to travel across diverse communities, discerning and respecting multiple perspectives, and grasping and following alternative norms.

Assessment of Literacies
The availability of an ever expanding wealth of information online, alongside the growth in the adoption of web enabled mobile platforms, impacts on how, when, and where we learn, who we learn with, and how we evaluate the legitimacy of information (Haythornthwaite & Andrews, 2011). Anywhere anytime learning now occurs with potential learners engaging with mentors, peers, and experts with minimal regard to time or place. These learner empowered collaborations complement recent research related to teaching and learning where social learning is viewed as the primary framework for effective pedagogical practice (J. S. Brown & Adler, 2008;Siemens, 2005 Government calls for education accountability through standardized testing has in part reinforced the concept of a 'back to basics' literacy approach. Standardized testing using more traditional assessment methods drastically limits the evaluation of the social and cultural skills and competencies associated with participatory skills. This point is emphasized by Mary Kalantzis and colleagues (2003) in noting that "accountability and commensurability has focused global attention on producing education outcomes which are simple to interpret, tangible and transparent, and easily comparable" (p. 15). There is a need to examine alternate assessment approaches that can complement calls for accountability and standardization against an imperative for implementing new approaches that can measure the full spectrum of multiliteracies. More recently, Literat (2014) noted that the few studies that have attempted to measure literacy have generally focused on an individual's understanding of text and or audio. Literat argued that alternate assessment methods are required to evaluate a broader suite of literacies.
Literat addressed this methodological problem by demonstrating the potential for more psychometric methods to provide potential indicators of new media literacy skills.
Literat's (2014) new media literacy questionnaire begins to illustrate the capacity for alternate approaches to identify potential lead indicators of literacy development and comprehension. As detailed in Table 1 many of the skills are not readily assessed through standardized testing. Literacies such as "play" or "negotiation" are socially embedded and enabled. As a result, effective assessment requires a more nuanced and diverse approach than is possible through basic knowledge testing.
Alongside the curricular adoption of the skills and competencies associated with multiliteracies, current assessment practices must extend beyond the measurement of outputs to provide greater insight into learning processes (Edwards, 1997). To facilitate this transition, David Boud (2000) maintained that assessment practices cannot remain the "exclusive domain of assessors" (p. 151) as responsibility also resides with the learner. It is only through an individual's capacity to evaluate, interpret, and make decisions about their learning progress will education foster the skills necessary for capable lifelong learners (Boud, 2000). Boud's notion of "sustainable assessment" and learner empowerment is in keeping with calls for the development of new assessment practices that address literacies such as creativity, networking, citizenship, and collaboration (Kalantzis et al., 2003). Thus, for education, contemporary assessment practice needs to reflect these community-centric learning models to include measures of learning progression at both the community and individual levels. This is exemplified in Csikszentmihalyi's (2006) call for creativity to be assessed at the level of community not as an end product derived from a sole individual. However, pressures of accountability, equity, standards, and quality assurance results in a loss of focus on assessment of community in lieu of an individual. It is at this intersection of education accountability and need for collaborative real-time assessment measures that analytics can provide an important role. The following section provides a brief overview of What are Learning Analytics?
The field of education has entered the era of "big data". The Mckinsey report (2011) defines "big data" as a "dataset whose size is beyond the ability of typical database software tools to capture, store, manage and analyze" (p. 1). As noted in the report, while the definition is subjective, the 'big data' concept relates to the flood of data that is generated and captured as users interact with the myriad of IT systems that support daily activities from iTunes, Twitter, and YouTube through to ecommerce and public services. In an education context, student information systems and learner interactions with various technologies such as learning management systems (LMS) and social media leave a trail of digital breadcrumbs that are accessible for data mining and analysis (Buckingham Shum & Ferguson, 2012;Duval, 2011;Fournier, Kop, & Sitlia, 2011;Macfadyen & Dawson, 2010). The extraction and analysis of the data derived from these technologies has captured the attention of politicians, education leaders, researchers, and day to day education practitioners. This is partly due to the expectation that these relatively new forms of, and processes for, analytics can address some of the more pressing concerns confronting the education sector. These concerns include increasing completion rates, addressing basic curriculum standards, student questionnaire overload, accountability and measuring teaching effectiveness and quality. Although to date the field of learning analytics has largely focused on learner progression and developing lead indicators of student attrition (Campbell, De Blois, & Oblinger, 2007;Fritz, 2011), there is much potential (and some early research) in using student interaction data to establish indicators of more complex concepts such as knowledge construction (Pozzi, Manca, Persico, & Sarti, 2007), sense of community (Dawson, 2006;Gasevic, Adesope, Joksimovic, & Kovanovic, submitted), creativity (Dawson, 2010;Dawson, McWilliam, & Tan, 2011), and self-regulated learning (Biswas, Jeong, Kinnebrew, Sulcer, & Roscoe, 2010;Winne & Hadwin, 2013).
Learning analytics adopts many of the methods and approaches used by "big data" practitioners, but is specifically defined as "the measurement, collection, analysis and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning and the environments in which it occurs" (Siemens & Long, 2011 The interest in learning analytics is fuelled by increasing calls for accountability, quest for evidence, and the demonstration of what counts as learning and teaching quality alongside decreasing fiscal resources. Similarly, the accessibility to student learning data from various information systems has also contributed to the use of learning analytics as a tool for measuring impact and informing the strategic decision making process within education organizations (Macfadyen & Dawson, 2012). Although the concept of using these forms of analytics for informing education practice has been well received, the majority of educational institutions seldom make optimal use of their available data and analytical resources (Norris, Baer, Leonard, Pugliese, & Lefrere, 2008). Furthermore, as a result of the relative infancy of the field of learning analytics, the level and sophistication of the data analysis performed has to date been limited, with a predominance of studies and reports undertaking simple univariate or bivariate analyses (Dawson, Gašević, Siemens, & Joksimovic, 2014). For instance, low-level analysis such as the reporting of student login times, number of posted messages in a discussion forum, or total time online are common analytics approaches. Bivariate analyses have tended to relate the readily accessible learning management system (LMS) data (e.g., number of messages posted, frequency of logins, time online, etc.) to academic performance as measured by grades.
These early studies have served well to raise the profile of learning analytics and to generate interest in the field as a means to improve student retention and success, including the development of early indicators of academic performance through LMS usage trends. The Purdue University course Signals project 2 is an example of early warning systems. The Signals software incorporates student past academic performance, demographics with LMS activity, and engagement to provide a statistical assessment of an individual learner's probability for success (Arnold, 2010). Watson and Gemin (2008) defined "at-risk" as students with a high probability of withdrawing from the course as a result of academic failure, dropping out, or expulsion for behavioral reasons. In the instance of Signals, success is measured by an individual's overall assessment score. The intent of the software is to provide early warning indicators to learners and instructors regarding an individual's risk of course failure or attrition.
The implementation of practices that focus on improving student retention has a direct and easily measurable return on investment and aligns well with numerous federal government initiatives internationally. For instance, the Australian Bradley Report Researchers such as Macfadyen and Dawson (2010) have investigated LMS data to predict student academic performance. While these authors used relatively simple metrics such as student grades as an indicator of success, the study incorporated a more sophisticated modeling process to analyze 15 LMS variables to determine and evaluate a best-fit predictive model. The authors note that social learning (such as in a discussion forum) and formative assessment tasks completed accounted for greater than 30% of the variation in student grade. Macfadyen and Dawson's study moved beyond the incorporation of simple engagement measures such as number of discussion messages posted towards more complex analytic measures such as social network analysis (SNA).
The inclusion of social network methods for determining community participation aligns with the move for new media literacies to be evaluated at the level of community (Jackson, 2006). Combining SNA with other more automated qualitative analytics through machine learning and computational linguistics can further enrich the prospects for establishing a meaningful model for evaluating new literacies. For example, while SNA provides an indication of the strength and diversity of relationships an individual actor establishes in a network there is minimal reference to be able to identify the purpose and value of this relationship. In this context, the inclusion of automated content analysis provides added insight that can determine the extent to which an individual demonstrates good participatory practice. More recently, learning analytics research has started to transition to more sophisticated methods targeting discourse, language, and affective learner attributes 3 .

Literacies and Learning Analytics
The inclusion of methods such as SNA, epistemic network analysis (Shaffer et al., 2009), and affective learner attributes (Baker, D'Mello, Rodrigo, & Graesser, 2010)  into the learner's competency for play and performance. For example, the degree to which the user engages with the resources, models, pathways, and specific goal oriented outcomes provides an indication of the user's efficacy for engaging in play activity as well as adopting alternate persona for the purposes of discovery, reflection, and perspective.

Products/Creation: Appropriation and Simulation
Products/creation can be observed directly through the generation of specific artifacts such as multimedia 4 . As such these products, associated methods for creation, and level of co-creation and engagement can be measured. In Axel Bruns' (2008) terms, the concept of prod-users defines a shift away from a model of production to a more collaborative and user-led model of creation. The concept reinforces the notion that any digital product can be remixed and repurposed, and as such is a continual state of flux and evolution. The feedback link from producer to consumer back to producer can be collaborative and completed within an extremely short time frame. Wikipedia illustrates this dynamic and evolving collaborative system. Essentially, the flow and utilization of products as they evolve in both social and cultural importance can act as an indicator of appropriation and simulation.
Network Agility and Citizenship: Collective Intelligence, Judgment, Negotiation, Distributed Cognition The skills grouped into network agility/citizenship represent an individual's level of competence and capacity for building relationships, participating in networks, and contributing to a community of learners. This cluster relates to a learner's role, position, and contributions to the learning network. This can be readily measured via social network analysis (SNA). The integration of SNA not only provides insight into the strength and diversity of relationships formed but also the types of information or resources shared within the social system (Haythornthwaite, 2002).
In examining the impact of network position and the flow of good ideas Ronald Burt (2004) illustrated the value of SNA to provide insights into network position and information and resource access. In this example, Burt noted that actors bridging two or more previously disparate network clusters demonstrate greater agility and enterprise than peers positioned within an insular network cluster. According to Burt (2004), these individuals "are able to see early, see more broadly, and translate information across groups" (p. 354). Burt sees this 'translating' function as value-adding creativity. This is not just because of the extent to which 'brokers' are able to move knowledge around in value-adding ways, but also their capacity to build, sustain, and expand upon their networks within and outside the existing environment. The skills these "border crossers" (McWilliam & Dawson, 2008)  The use of content analysis alongside SNA affords rich insights into the quantitative as well as qualitative aspects of a social system. Content analysis is a commonly adopted approach for determining the perceived quality of the knowledge construction process occurring within a learning network (De Laat, 2002;De Laat, Lally, Lipponen, & Simons, 2006). However, the mapping of the captured exchanges to a pre-defined coding scheme has to date been largely a manual and time consuming process. Hence, the level of integration between SNA and content analysis methods has thus far been minimal at best. The adoption of machine learning techniques and tools such as TagHelper (Rose et al., 2007)   Debate, discourse tools, give and take interactions in knowledge development (Pozzi et al., 2007) The media literacies described are still reliant upon an individual student demonstrating a certain level of competency in the more traditional literacies (Jenkins et al., 2006). A student who has limited reading and writing proficiency will continue to struggle with new media literacies. As depicted in Figure 1, the traditional literacies are the foundation for all other literacies. These basics can be evaluated through automated content analysis. Textual passages can be extracted from student activity with blogs, wikis, traditional writing assignments, discussion forum activity, even Twitter posts.
However, while this data can be mined and analysed, an understanding of the learning design is essential for establishing meaningful indicators and assessment of an individual's proficiency within one or more literacy (Lockyer, Heathcote, & Dawson, 2013).
Learning analytics and educational data mining frameworks have been created by researchers to describe the range of activity, in terms of techniques and applications, undertaken by researchers working with learning-related data (Baker & Yacef, 2009;Bienkowski, Feng, & Means, 2012;Siemens, 2013). These suggested frameworks reflect a sequential maturing and refinement of analytics work whereby the more recent work undertaken by Siemens incorporates and builds upon the earlier studies. Siemens'  Figure 1), experimentation, products/creation, network agility and citizenship, and task effectiveness and efficiency, to an existing framework of learning analytics techniques and applications (Siemens, 2013) can provide the analytics opportunities as described in Table 3.

Conclusion
With growing interest in data and analytics in the education sector, it is important for researchers, educators, learners, and administrators to have tools and techniques that go beyond surface-level analytics. The complexity of a social process in learning cannot be adequately assessed through basic metrics such as logins, time online, and clicks.
Multiliteracies draw attention to the skills and attributes learners require to navigate the increasingly complex technical, social, cultural, and economic worlds. However, traditional models of standardized assessment do little to either promote or effectively measure these multiliteracies (I. Brown et al., 2010;Kalantzis et al., 2003).
This paper builds upon established theoretical models to demonstrate the role learning analytics can play in assisting educators and students in developing real-time feedback and analytics for evaluating literacies. Thus, the significance of the model resides in its capacity to provide deep and nuanced insight into the learning activities of students and to merge the boundary between multiliteracies and learning analytics. In this context, alternate and diverse assessment techniques and instruments are necessary to better align and reflect the technical and information complexity and multimodal learning that form the core of 21 st century education.