Open Assessment of Learning: A Meta-Synthesis

Open Assessment of Learning (OAoL) is an emerging educational concept derived from the incorporation of Information and Communication Technologies (ICT) to education and is related with the Open Education Movement. In order to improve understanding of OAoL a literature review was conducted as a meta-synthesis of 100 studies on ICT-based assessment published from 1995 to 2015, selected from well-established peer-reviewed databases. The purpose of this study focused on identifying the common topics between ICT-based assessment and OAoL which is considered as an Open Educational Practice. The review showed that extensive use of the Internet makes it easy to achieve some special features of OAoL as collaboration or sharing, which are considered negative or inconvenient in traditional assessment but at the same time become elements that promote innovation on that topic. It was also found that there is still a great resistance to accept change (as OAoL does) when structural elements of traditional assessment are questioned or challenged.


Introduction
Assessment of learning has been considered for a long time a fundamental component of teaching and learning processes (Brailovsky, 2001;Brown, Bull, & Pendlebury, 2013;Dochy, Segers, & Dierick, 2002) and at the same time, it has received some of the harshest criticism from different educational stakeholders (Beaumont, O'Doherty, & Shannon, 2011;Wiliam, 2011).
The specialized literature on student assessment is abundant and profuse and has shown over the years a variety of diverse conceptualizations and applications but usually with the same meeting point: the assessment as the measuring of learning goals or outcomes, generally associated with knowledge acquisition, standardized test-based or grade-oriented (Astin & Lising, 2012).It should be noted that having some historical exceptions on the above, as the approach of John Dewey in early 1990s on the "learning by doing," where a formative or alternative assessment was promoted, (now becoming an emerging practice in some developed countries), the "classic" way to assess learning still remains in a lot of educational contexts around the world (Kamens & McNeely, 2010).
Moreover, the impact of massive use of the Internet has been manifested through remarkable changes in the way we communicate and relate to others (Joinson, 2003).Education is one of the areas that have been affected by this, but not in the same way or with the same intensity as, for example, trading, entertainment, or communication.Although over the last three decades, education has shown some interesting ICT-based changes in its dynamics (e.g., the emergence of e-learning, the use of mobile devices and more recently some open learning experiences known as massive open online courses or MOOCs), these examples are not yet very signifcant in the overall panorama of education (Livingstone, 2012;Yildiz et al., 2014).
As a result, the use of the Internet is creating incipient changes in teaching practice or the way in which educational content is produced or distributed (Gillespie, 2014;Pera, 2013).However, assessment persists in repeating the same mechanical models that hinder the transition from a summative to a formative type of assessment, changing only the means and tools to technological resources but the nature of learning assessment remains the same (Redecker & Johannessen, 2013;Voogt, Knezek, Cox, Knezek, & ten Brummelhuis, 2013).
Moreover, as a revitalization of early 20th century Open Education, a growing trend is emerging called an "Open Education Movement" that is gaining ground from an extensive use of ICT.This movement promotes a way of thinking about education that transcends its origins that were linked to distance education and currently configure a set of open educational practices that enhance teaching and learning by incorporating "openness" attributes such as sharing, remote collaboraton, adaptation, free access, and reuse of information and spaces for educational interaction, etc. (Downes, 2013;Knox, 2013).
Regarding the above, it becomes interesting to think about a new or at least a fresh concept for student assessment called Open Assessment of Learning (OAoL), understood as an open educational practice and defined as follows: The process of learning verification and feedback that takes place collaboratively, mediated by free access tools in which teachers produce or adapt assessment resources and students adapt and reshape these resources for the purpose of generating for themselves an assessment that meets their personal needs, learning styles and context.(Chiappe, 2012, p. 10).
Therefore, as OAoL looks to be a promising or at least provocative concept, which aims to promote innovative changes in educational practices, it becomes relevant to ask this question: Conceiving OAoL as an ICT-based educational practice, which are common topics among ICT-based assessment and OAoL so that it can be better understood as an open educational practice?
This question led a literature review on ICT-based student assessment, which is the context in which OAoL is developing and will be the scenario of its implementation as an open educational practice.

Method
The review of literature is a very particular type of study, characterized as a reflexive process of social inquiry that is based on the analysis of written scientific research (Ahmed, 2010;Denscombe, 2014).
This study used meta-synthesis as a method that differs from the meta-analysis and the systematic literature review and focuses on the study of the results of other qualitative research in order to generate substantive theory through interpretation of such results (Thorne, Jensen, Kearney, Noblit, & Sandelowski, 2004;Walsh & Downe, 2005;Whittemore & Knafl, 2005;Zimmer, 2006).The analytical and selective review of the literature for the purpose of this study was developed as a Content Analysis process, which was based on the observation of evidence of key subjects or topics that would favor or limit the implementation of OAoL.In support of this method, Sandelowski, Docherty, and Emden (1997, p. 369) write, "in contrast to quantitative metaanalysis, qualitative metasynthesis is not about averaging or reducing findings to a common metric, but rather enlarging the interpretive possibilities of findings and constructing larger narratives or general theories." The review was implemented using three steps based on the basic review cycle of Machi & McEvoy (2012), as follows Step 1: Establish the purpose of the review (define research interest).As previously mentioned, the main objective of the review was to identify in peer reviewed research papers, some common topics among ICT-based assessment and OAoL.This would allow the researchers to progress towards the understanding of the particularities of OAoL as an open educational practice.
Step 2: Specify inclusion and exclusion criteria and explore a concept in multiple sources (searching and filtering).The first inclusion criteria had to do with ensuring the quality, diversity, and amplitude of information sources.For this reason, the initial searching was conducted in well-recognized peer-reviewed academic sources that allow access to papers in English, Spanish, and Portuguese.Thus, the searching was conducted through ISI Web of Knowledge, Scopus, Scielo, and DOAJ, which allowed reflecting on European (66%), North American (15%), and Latin American (4%) thought on the subject.
Second inclusion criteria had to do with searching relevance.To do so, searching descriptors or keywords included: "open assessment," "assessment + ICT," "assessment in virtual learning environments," and "assessment + elearning."All descriptors were applied both in English, Spanish and Portuguese.
After the first search in databases, we excluded letters, erratums, editorials, notes, shorts surveys, books, and book chapters.Only reviews, research articles, and conference papers on Social Sciences were selected for further analysis.
A final inclusion criteria had to do with date filtering.Because we were looking for research related to ICT-based learning assessment, only papers published between 1995 to 2015 (note that the first paper was found on 1998) were finally selected.
As a result of this first process, a set of documents with 664 items was created.Figure 1 shows date distribution of this first set of documents.The last filtering was carried out through an abstracting proccess that allowed reducing the document set to a final group of 100 items, ready for further in-depth reading as shown in table 1. Step 3: In-depth reading and data processing.Based on the proposal of Guba and Lincoln (2000) on the relevance of conducting an "audit research" in which contributors examine both the processes and consistency in research products, an independent and parallel two-observer reading was conducted on the same set of documents.
As shown in Figure 2, data processing starts with an in-depth independent two-reader reading and extraction of relevant subjects or concepts in the form of small chunks of text arranged in separate lists.
In order to strengthen the objectivity and reduce the level of bias in the analysis, a Cohen's Kappa coefficient (K) was applied to the separate lists that contain text segments analyzed by each reader.The result of this process was K = 0.74, which corresponds to a high level of consistency.
Then, those two lists containing the text segments and their associated concepts or key ideas were unified into a single document, which was processed and compared through a text mining tool as mentioned by Botta-Ferret & Cabrera-Gato (2007), which allow to identify information patterns within the data.These patterns were converted later into sets of key issues, which ultimately became sets of results of the review.

Results
The literature review process on ICT-based student assessment shows that the intellectual production and research in this area has been extensive and varied, since the mid-90s.As said before, 100 peerreviewed research papers were analyzed.The review showed a large number of subjects potentially identifiable with the conceptualization proposed for OAoL.In this regard, and for descriptive purposes, the issues were grouped into six topics, which are described below.

The Teacher-Focused Assessment
Much of the literature, corresponding to 63% of analyzed documents regarding assessment with ICT reveals one of the main features that simultaneously includes its deepest criticism: the teacher's dominance and the learner's passivity when engaged in assessment and identification of their own learning.For decades, international agencies such as United Nations Educational, Scientific and Cultural Organization (UNESCO) and the Organization for Economic Co-operation and Development (OECD) have suggested to governments and institutions new approaches to teaching and learning in order to foster quality education as a means to contribute to the eradication of poverty, taking into account factors of coverage, timeliness, easiness, and relevance.In other words, to democratize education (Semenov, 2005) and change the paradigms of educational methods centered on teaching to a student-centered instructional learning in which the assessment acquires a new dimension (Mainali & Heck, 2015;Weimer, 2002).This new dimension makes allusion mainly to a change from student's passive role to an empowerment that involves active participation in assessment design and implementation that will ultimately lead to the creation of customized assessment experiences.
Also, 53% of such authors agree with these views and support a transformative discourse based on the use of ICT, especially focused towards its potential to transform personal access to information, to spaces of social interaction and the strengthening of digital skills and autonomy.In fact, 65% of such papers refer to ICT-based assessment as a diametrically opposed option to one that is focused and controlled by the teacher, even if teachers have acquired competence on teaching with ICT (Angeli & Valanides, 2009;Bottino, 2004;Cobcroft, Towers, Smith, & Bruns, 2006;Richards, 2005;Tinio, 2003;Voogt, 2003;Wang, 2008).Perspectives include that of technological pedagogical content knowledge (TPACK), the articulation of pedagogy, technology, and learning needs; the use of emerging technologies; the changes in the role of teacher; and challenges facing developing countries.
However, daily assessment practices, at least for the most part, are not echoing these ideas and, on the contrary, reinforce traditional patterns of teaching and assessment (Mora & Romero, 2014;Zu-wang, 2007).In such traditional practices the learner is considered a passive receiving agent, absent of criterion and experience in order to be able to empower their learning and, of course, their assessment (Montgomery, 2002;Sitthiworachart & Joy, 2008).Scenarios like this establish a clear context of reluctance to embrace a potentially disruptive concept such as Open Assessment of Learning.
This has negative implications for short-term participatory assessment processes that are very characteristic of OAoL.A teacher-centered assessment limits the implementation of collaborative or participative assessment and therefore restricts the possibilities of using systems or methods that provide learning assessment as a product of teacher-student consensus.As Dos Santos (2013, p. 125) stated when referring to education (which also applies to the assessment based on this perspective): It is not just a question of a teacher acting as a promoter of someone else's learning, but is also about someone who is also learning, mediated by the group that questions, comments and reacts to the different stimuli we bring.
In this situation, by not engaging in OAoL, both the teacher and the student suffer by letting opportunities to learn together or from each other pass them by.

OAoL Turns Assessment from Summative to Formative
Another recurring issue that appears in the literature presents critical reflections regarding the assessment as a process that identifies the levels of knowledge and skills acquisition.Forty percent of the analyzed texts consistently show the call for transit from a summative to a formative perspective of assessment and the focus on the use of ICT to evaluate both people and programs (Rodriguez, 2005).
This concept of assessment certainly is not new and has been widely studied for a long time, by authors who have addressed this from the pedagogical discussion, the use of mobile devices, the importance of motivation in formative assessment and learning analytics (Black & Wiliam, 2009;Hwang & Chang, 2011;Nicol & Macfarlane-Dick, 2006;Tempelaar, Heck, Cuypers, van der Kooij, & van de Vrie, 2013;Yorke, 2003).This is certainly not only a favourable condition for the development of Open Assessment of Learning but one of its features that allow us to think that this kind of open educational practice facilitates the transformation of strictly summative assessment into a formative one.This involves making assessment as a learning opportunity, not just to identify gained learning but also to open the possibility of conceiving other realities therein, in which, for example, assessment processes do not need observants or where items or practices that are not allowed in traditional assessments are now well received, such as collaboration and consultation becoming desirable to achieve the objective of strengthening the learning of those being assessed.
In this sense, it is possible to consider the assessment as a process of self-regulation as expressed by Valdivia (2009).This recogniszes the importance and formative potential of the timely socialization between teachers and students as well as the participatory development of assessment tools and the cocreation of customized assessment experiences.In addition, the application of the attributes of "openness" lead to a less guided-by the-teacher participation, which helps the generation of digital communities of learning that require the strengthening of student autonomy (Willis, 2011).The student must make decisions based on their criteria and not from a direct, and predetermined orientation by the teacher, which in turn requires greater self-regulation by the student.

The Internet: More Than a Tool, a Transformation Factor
Open Assessment of Learning has an underlyng transformative nature in the way it incorporates ICT.
It is worth mentioning that the Internet is just a technological tool and as such, the interaction dynamics that are built based on this (collaboration, self-assessment) will not have the desired effect if the learning experiences have not previously been designed with this in mind.In this regard, OAoL proposes an enabling environment for harnessing the transformative potential of the Internet.
In this sense, the literature does not make any special emphasis on this condition but identifies the role of the Internet in assessment processes as a supporting component, mediator, or as an assessment Given the complex challenge of further developing their transformative potential, it is reasonable to assume that it is easier to conduct an instrumental approach to ICT, which is perhaps one of the main reasons for this emphasis in the specialized literature.However, 73% of the authors that addressed this issue also identified or proposed two elements that might eventually lead to generating changes in assessment practices via the use of the Internet and that can be included as part of OAoL: digital rubrics and e-portfolios.
It should be noted that although these tools are not really new or innovative in the context of the ICTbased assessment, they become complementary to what has been mentioned in the previous section regarding the strengthening of active participation of students in their own assessment process.In fact, under the category of digital rubrics, assessment processes using web systems known as "e-rubrics" are used in naturally complex interdisciplinary and collaborative learning environments.Some examples of this were implemented to assess various tasks to be performed in forums (Bartolomé, Martínez-Figueira, & Tellado-González, 2014;Torres-Gordillo & Perera-Rodríguez, 2010), peer review processes (Serrano & Cebrian, 2011), or in co-peer reviews and self-assessment (Gallego-Arrufat & Raposo-Rivas,

2014).
As suggested by Panadero and Jonsson (2013), the use of digital rubrics provide transparent assessment, which also helps to reduce anxiety in students caused by such process.Moreover, the use of rubrics can also facilitate feedback and improve self-efficacy and self-regulation of learning, all of which can indirectly facilitate the improvement of student achievement.
In addition to digital rubrics, Cebrián de la Serna (2011) indicates that e-portfolios provide support to both formative assessment and constructivist approaches to teaching and learning (and in this case also in assessment), in which three key elements that contribute to this process are present: reflective practice, collaborative learning, and peer-teacher feedback (de la Cruz Flores & Abreu Hernández, 2014).

The Importance of Assessment Feedback
Feedback is another major element identified in ICT-based student assessment, which is emphasized as an enabling factor that should be considered beyond just quantification or measurement of learning.
The literature review on this subject is extensive and details the educational importance for the students of receiving timely and relevant information about their learning, the perspective of automated systems, the possibilities of tailored and timely feedback and its importance to maintain adequate levels of student motivation as well as the relevance of peer feedback (Debuse, Lawley, & Shibl, 2007;Gipps, 2005;Irons, 2007;Jordan & Mitchell, 2009;Li, Liu, & Steckelberg, 2010;Van den Berg, Admiraal, & Pilot, 2006;Webb, 2005).
According to Simpson (2013), feedback should consider the emotional aspects of the student because this increases motivation, develops new and better thinking and learning skills, and reduces student dropout rates.Here arises the question of whether or not the teacher or tutor will have the skills and knowledge to achieve such monitoring and feedback in a highly ICT-mediated context.Some authors state that it is possible, but requires a great deal of teacher commitment, time, and training (Hatziapostolou & Paraskakis, 2010).
From this perspective, an approach to assessment as an open practice seeks to improve the participation of the learner and their centrality in the feedback process as a key actor -as peer learner-, encouraging commitment to learning and increasing the validity and reliability of assessment activities through the implementation of actions and ongoing formative feedback through Internet tools (Gikandi, Morrow, & Davis, 2011).
OAoL, due to its collaborative and participatory features, provides feedback from different directions and unexpected content, which might introduce different and valuable information to feedback that may even exceed the teacher's single point of view about feedback content.

Regulation and Status Quo: Major Barriers for OAoL
The literature review draws attention to a particularly limiting condition that affects OAoL.Twenty-six percent of analyzed papers agree that current education is ruled by assessment traditions that are very difficult to change and that are part of what is considered status quo.Reviewed papers show that this phenomenon is not focused on a particular geographic region.Although there are exceptions, traditional assessment is a globally widespread reality.
In addition to this cultural limitations, there are institutional and governmental rules and regulations that increasingly reaffirm this tradition (Chetwynd & Dobbyn, 2011).Standardized tests and international benchmarks, such as the Programme for International Student Assessment (PISA), eventually become governmental standards or reference points, which in turn are trickled down to educational institutions and into classroom practices.Any alternative to the established norm automatically enters unknown territories and is therefore "risky and dangerous", which is one of the main ideas that difficult change in the traditional evaluation.(Boud, Cohen, & Sampson, 1999;Heller, Watson, Komar, Min, & Perunovic, 2007;Maslovaty & Kuzi, 2002;Stoiber & Vanderwood, 2008).
In addition to the above, literature shows an emerging attempt of shifting the traditional concept of assessment, moving from a selective and quantitative conceptualization of assessment to a more flexible and participatory approach.However, this change is not yet reflected in concrete and coherent actions at the insitutional level, which is identified as an unfavorable factor for implementing innovative conceptions of ICT-based assessment such as OAoL.

Assessment Procedures
Ninety percent of the texts analyzed in this literature review put particular emphasis on the relevance of applying different forms of assessment (self-and co-assessment or peer assessment) in addition to assessment conducted by teachers (Sitzmann, Ely, Brown, & Bauer, 2010).This is particularly relevant for OAoL due to its technological context.Literature shows that the more technological mediation, the greater the presence of peer and self-assessment, which becomes a very interesting subject for instructional and assessment designers of blended, elearning, or even MOOC learning experiences (Barry, 2012;Budimac, Putnik, Ivanović, Bothe, & Schuetzler, 2011;Chang, Tseng, & Lou, 2012;De Grez, Valcke, & Roozen, 2012;Gielen, Dochy, & Onghena, 2011;Lai & Ng, 2011;Shih, 2011).
Regarding the above, peer and self-assessment are valuable components for OAoL because they introduce to the learners the necessary ethical values of responsibility and honesty, throughout the whole assessment exercise, allowing to conceive it in a way that is farther from grading and closer to the strengthening of learning.This creates a context of mutual trust, where the teacher´s role in avoiding "cheating" is no longer needed, so he can focus his energy to support and promote student´s learning via both teacher´s and peer´s feedback.
In that sense, Castillo-Merino and Serradell-López (2014) propose consultative assessment processes so that there are collective constructions that contribute not just to the strengthening of motivation, which is of great importance, but also to extend the exercise to the metacognitive generation driven by the rewards of assessment in collaboration.This can be carried out in the collaborative construction of rubrics and assessment tools in which teacher and students work in consensus to involve, for example, student's learning styles and their preferences for the type and conditions for assessment.

Discussion
It could be said that the boundaries that have traditionally been applied to a "class" for students are disappearing because of the integration of the Internet into academic dynamics of educational institutions.Current conception of a "class" for many of today's students includes an increasingly ICTbased complex network of interactions (Hartman, Dziuban, & Brophy-Ellison, 2007).If this is true, within such complexity traditional assessment offers a poor opportunity for interaction.Perhaps we should consider new protocols or new assessment methods leading to spaces of greater reflection and interaction.
In addition to the above, and recognizing that OAoL is a process based on the use of ICT, the results show that the technology should not be as important as the application of the attributes of "openness," which would not have the same scope if they are out of their digital ecology.In that vein, it should be recognized that technology is a tool that can generate transformation or simply become a distraction factor for innovation, making things the same way but with new mediations.That's why it makes sense to consider that is how to assess with technology, which would lead to generate transformation in such educational practice.In this context, the application of the attributes of "openness" start to generate these new possibilities to think differently about assessment, and then, to assess differently.
OAoL, then, becomes an alternative to traditional forms of assessment.Without a doubt, the implementation of this open educational practice would significantly increase participation and empowerment of the students in their learning and assessment.In this regard, it is noteworthy that transformation of the traditional assessment culture will be an arduous task; a process that will require a significant investment of energy, time, and resources.
Thus, it is expected that OAoL experiences must produce an emerging transformation in assessment, and although it was not intentional, a transformation in teaching practices.This is interesting as a potential innovation strategy that seeks to break resistance to change from teachers that are engaged in traditional educational practices.
Moreover, the transformation of assessment into a more open process, conducted under the application of some of "openness" attributes, makes the assessment more formative and less teacher-centered and puts the focus out of the mere measure of learning outcomes.Thus, assessment becomes a moment to learn, not just the space in which the learner is accountable for their learning.
Besides the above, it is important to note that OAoL should not be seen as a panacea in terms of student assessment.While it may be considered a new practice, it is prudent to consider it as complementary to other forms or types of assessment.It should be understood that any learning assessment deals with people with distinct expectations, interests, learning styles, backgrounds, educational contexts, and availability in terms of time and space.As a result, using multiple mechanisms to identify and enhance learning should not be discounted.It should also be recognized that the presence of OAoL does not imply the complete elimination of traditional assessment, which can begin to transform itself, preserving its valuable components and complementing them with open strategies, better resources, more flexibility of time constraints, and above all, follow the discourse that the student is the central axis of any real learning process.
Finally, "openness" puts us in a place where different elements traditionally considered inappropriate or unacceptable become valid or even desirable in the educational scenario.This is why it is necessary to change the way of thinking about assessment that, by the way, creates major challenges both institutional and personal.In that sense, an educational institution that considers implementing open assessment practices will be in permanent tension with an educational system that is not used to it and that works within a normative framework that is inconsistent with the nature of the OAoL.Moreover, OAoL presents personal challenges both for teachers and students to the extent that the first must be willing to cede control that traditionally have on assessment, and students would be able to change the idea about that assessment is equal to grading or even promotion instead to recognize that really is a space to keep learning about themselves and from others.

Prospective and Limitations
Derived from the findings of this study, some interesting topics for future research appeared.One of them has to do with a possible relationship between compromise generated by OAoL empowerment and decreasing drop-out rates or strengthening student's academic success.
One of the limitations of this study that merits further research efforts has to do with the contribution of OAoL to the management of the permanent tension between the need for educational innovation and the responsibility that educational institutions face to ensure their quality.In fact, one of the immediate problems of OAoL is the incompatibility with some of the assessment methods (accepted by education systems) that educational institutions have traditionally used to ensure the quality of their graduates.
In this sense, OAoL could be implemented progressively and complementary to traditional assessment methods through multiple-case research projects.This would generate sufficient institutional experiences to validate the relevance of OAoL as a fully reliable assessment alternative.

Figure 1 .
Figure 1.First selection of documents by year.

Figure 2 .
Figure 2. Detail of in-depth reading and data processing.

Table 1
Final Set of 100 Items for In-Depth Reading