Volume 18, Number 5
Gema Santos-Hermosa1, Núria Ferran-Ferrer2, and Ernest Abadal3
1,3University of Barcelona, 1,2Open University of Catalonia
This article provides an overview of the current state of repositories of open educational resources (ROER) in higher education at international level. It analyses a series of educational indicators to determine whether ROER can meet the specific needs of the education context, and to clarify understanding of the reuse of open educational resources (OER) provided by ROER. The aim of the study is to assess ROER by combining these two perspectives, and to form a basis for discussion among the universities that are responsible for these repositories.
The method was based on content analysis and consisted of two phases: an exploration of international sources, and an analysis of 110 ROER using the proposed set of indicators. The results focus on data from the analysis of ROER websites and some models of good practices. They are presented according to three core dimensions for evaluating ROER: general factors to establish types of ROER, a focus on drivers for OER reuse, and a focus on educational aspects. It was found that most of the ROER that included one or more of the proposed reuse indicators were created exclusively for educational resources. Educational aspects are not yet firmly embedded into ROER. The few repositories that seem to have successfully included them are those that provide other educational metadata and use educational standards.
Keywords: repositories, open educational resources, OER, evaluation, reuse, higher education
In the past decade, the Open Educational Resources (OER) movement has gained ground and expanded exponentially. Previously, there were only localized OER initiatives; now OER are recognized and supported by key international institutions, governments and funders worldwide. Although OER have become more significant worldwide and their availability and use have expanded, the movement has still not achieved its full potential and entered the mainstream education system (Ochoa & Duval, 2009; Dickev & Dicheva, 2012; European Commission, 2013a).
In February 2015, the collaborative report "OER Strategy Development" (Allen, Browne, Forward, Green, & Tarkowski, 2015) provided a concise analysis of the current state of the OER movement and became a roadmap to identify specific strategies to achieve the "real adoption" of OER (understood as taking advantage of the rights and permissions granted by OER for use and reuse).
According to the report, current platforms that enable the management, discovery, use, and reuse of open content are inadequate and not very well-known; resources are found in a wide range of formats, and repositories are varied and generally do not include common search terms or metadata. For this reason, a study on the extent to which existing OER repositories (ROER) promote the discovery, use, and reuse of resources may lead to improvements in the adoption and impact of OER.
The OER movement in Europe has led many universities to rethink their institutional policies, and consider how to innovate in higher education teaching and learning practices. In its Opening up Education initiative, the European Commission (2013a) observed that "stimulating supply and demand for high-quality European OERs is essential for modernizing education" and that "OER must become more visible and accessible to all citizens." One of the key transformative actions proposed is to improve the visibility of quality OER produced in the EU by 2020, through the development of open digital ROER using investment funds.
Evidence (European Commission, 2013b) shows that greater exploitation of OER would increase access to education, and the development of quality educational standards would have a positive effect. Consequently, ROER should include educational aspects and metadata so that the most relevant resources are classified and can be retrieved by the main users: teachers and students.
This study takes into account a series of educational indicators that determine whether ROER can meet the specific needs of the education context. It also defines the scope of ROER and proposes a series of indicators to analyze them, and particularly to clarify understanding about the reuse of OER provided by ROER. Consequently, this study could help to determine what can be done to improve ROER and may become a basis for discussion among universities and institutions responsible for these repositories. This is deemed essential, since ROER can make a substantial contribution towards achieving the goals of the Europe 2020 strategy: quality and relevance in higher education (European Commission, 2014).
Global knowledge and research in OER is increasing. This topic has aroused the interest of researchers, as OER could make a strong contribution to improving education and fostering openness.
One of the main themes of OER research is OER and technology, and repositories in particular (Zancanaro, Todesco, & Ramos, 2015). A pioneering, complete overview of ROER that examined their main features was conducted by Tzikopoulos, Manouselis, & Vuorikari (2007). Several other studies focused on guidelines for developing ROER (Organisation for Economic Co-Operation and Development [OECD], 2007; UNESCO, 2011), their structure (McGreal, 2011), operational aspects and a quantitative analysis of repositories (Ochoa & Duval, 2009), additional services that they might provide (Zervas, Alifragkis, & Sampson, 2014; de los Arcos, Farrow, Perryman, Pitt, & Weller, 2014) and a holistic approach to quality assurance (Atenas & Havemann, 2013; Clements, Palowski, & Manouselis, 2015). Similarly, a recent study focused on the institutional promotion of OER through repositories (Castaño, Punie, Inamorato, Mitic, & Morais, 2016) and details of ROER have been gathered in international directories, such as the OER World Map (D'Antoni, 2013), OER Repositories World Map (Atenas & Havemann, 2014) and OER Atlas (Neumann & Muuß-Merholz, 2016).
However, none of these studies specifically refers to the reuse and educational aspects of ROER: these two unresolved issues merit particular attention in our research.
Wiley (2000) considered that any digital resource could be reused to support learning, and subsequent studies have stressed the importance of reuse. However, they have also recognized the great difficulty in finding evidence of reuse (Orr, Rimini, & Van Damme, 2015), the still insufficient analyses (de los Arcos et al., 2014), and the "problem of remix" (Wiley, Bliss, & McEwen, 2014). The most common discussion framework has been focused on user surveys (McKerlich, Ives, & McGreal, 2013; Camilleri, Ehlers, & Pawlowski, 2014) and tracking OER usage (Pegler, 2011). A couple of studies specifically address the connection between reuse and ROER as a key theme that represents one type of activity within a repository (Atenas & Havemann, 2013), or link reuse to the creation of personal spaces (Cohen, Reisman, & Sperling, 2015). Nevertheless, ROER's current "predisposition" to reuse has not been considered. For this reason, the present research examines opportunities provided by ROER that facilitate reuse of OER.
Regarding educational issues, studies have mainly focused on benefits, challenges, and limitations associated with OER (Wiley et al., 2014), their creation and distribution (open textbooks, repositories, OCW courses and MOOCs), and their important role in open education (Lane & McAndrew, 2010; UNESCO, 2011; OECD, 2015). There are many studies on metadata evaluations of ROER (Bueno-de-la-Fuente, Hernández-Pérez, Rodríguez-Mateos, Méndez-Rodriguez, & Martín-Galán, 2009), but these adopt a formal and technological perspective rather than analyze the educational relevance of ROER (Rodríguez, Dodero & Sánchez-Alonso, 2011; Atenas & Havemann, 2013). Therefore, the current study aims to explore how ROER support teaching and learning, based on educational requirements.
We are aware of the connection between reuse and educational aspects, and understand that reuse can be more difficult because the educational design is rarely visible (Wiley et al., 2014). Therefore, this is the first review that integrates both perspectives in ROER. It also provides an overview of the current state of ROER in higher education.
As some aspects of ROER are underdeveloped, a distinctive, value-added framework is becoming indispensable. Since reuse and educational aspects in ROER still pose certain challenges, we propose the following research questions:
The method used in this study was based on content analysis (Krippendorff, 1980) and consisted of two phases:
In order to select ROER for analysis, the following sources were used: the Registry of Open Access Repository (ROAR), the Directory of Open Access Repositories (Open DOAR), the portal Open Education Europe, a report on the State of the Art by the Educational Repositories Network (EdReNe), and other literature on the subject.
After retrieving a total of 1,186 repositories of "learning objects" or "educational resources," a population of 110 ROER was selected using the following criteria (Figure 1):
Figure 1. Selection of the ROER population (Santos-Hermosa, Ferran-Ferrer, & Abadal, 2015).
Repositories tend to be evaluated using institutional factors that are based on institutional needs and strategies (Westell, 2006; Serrano, Melero & Abadal, 2014). However, considering that ROER's particular requirements differ from those of traditional digital repositories (Cervone, 2011), other aspects could be taken into account.
The current study proposes three core dimensions for evaluating ROER: general/descriptive factors to establish types of ROER, a focus on drivers for OER reuse, and a focus on educational aspects. These categories are broken down according to specific indicators.
Category 1: General indicators: Types and technology.
Table 1
ROER Types and Technology
Category 2: Indicators of reuse. One of the benefits of ROER is that teachers have the opportunity to "reuse, modify, and adapt resources" (Hylén, Damme, Mulder, & D'Antoni, 2012), and a secure and reusable platform is an important design factor that contributes to reuse (Wills & Pegler, 2016). Hence, ROER that implement drivers contributing to reuse would become "beneficial reusable platforms" for users. Table 2 shows a proposal of facilitating factors, and gives reasons for this chosen focus.
Table 2
Indicators for Reuse
Category 3: Educational indicators. Since it has been emphasized that the pedagogical value of educational resources is the most important factor (McGreal, 2004), Table 3 proposes a series of indicators that are designed to describe ROER from an educational perspective.
Table 3
Educational Indicators
Content analysis was carried out for each of the 110 repository websites, using Tables 1 to 3 as codification frameworks. Tools for retrieving information from each website were used, such as searching (filters and metadata) and browsing (documentation). Data were collected between 2015 and 2016.
Content analysis limitations were broken URLs, duplicate data and the omission of pertinent data. Another limitation of this study was that some repositories are available in other languages, and have not been translated into English.
This section presents and discusses data from the analysis of ROER websites. In addition, some relevant ROER are highlighted as models of good practices.
Main features. Table 4 indicates that there are more OER repositories that only contain learning resources (54.5%) than hybrid repositories (45.5%) containing research and learning resources.
Table 4
OER Stored Wholly or Partially
Most ROER (Tables 5 and 6) were multidisciplinary (73.6%). The second most common discipline in ROER was Science, Technology, Engineering, and Maths (STEM, 12.7%). Most ROER were dedicated to higher education (85.5%); a lower number covered different levels of education (14.5%). Although a multidisciplinary approach also seemed more common in a previous analysis (OECD, 2007), other data suggest that single-subject repositories would be more suitable for teachers, since it is harder for teachers to find and use relevant OER for their subjects in broader repositories (Sampson, Zervas, & Sotiriou, 2013).
Table 5
Disciplines
Table 6
Educational Level
Authorship. It was found that the prevailing ROER were institutional (76.4% created by universities, foundations or institutes) and the rest were national (23.6% funded by governments, national organizations or consortia). This pattern is similar to that found in another analysis (Atenas & Havemann, 2013), in which institutional ROER were also predominant, but to a lesser extent (50% of the total, while the rest was broken down into different types).
Geographical origin. As stated in the literature (Tzikopoulos et al., 2007; Atenas & Havemann, 2013), most ROER were created in Europe (72.7%) and North America (16.3%) (see Table 7). Others were scattered between South America (6.3%), Asia (3.6%) and Africa (0.9%). Twenty-seven countries were represented in the total ROER population (Table 8). The countries that created the highest numbers of ROER were the UK (18.1%), Spain (15.4%), the USA (13.6%) and France (10%).
The success of ROER in the UK might be due to the considerable funding provided for OER projects by the Joint Information Systems Committee (JISC). This pattern varies in other studies (Bueno-de-la-Fuente et al., 2009), in which the USA was found to be the most common location (28%), followed by Germany and Italy. However, our results might be influenced by the fact that they only focus on higher education ROER. The countries that were found to be most involved in ROER may coincide with the countries in which most publications about OER have been identified (Zancanaro et al., 2015).
Table 7
Geographical Origin of ROER (Continents)
Table 8
Geographical Origin of ROER (Countries)
Software platform. Data on the technology that supports ROERs platforms (Table 9) suggests that DSpace is the most popular software (34.5%), followed by a range of other software (19.1%), some content management systems (CMS) and learning management systems (LMS) (11.8%), and Eprints (7.3%). Other software that is used to a lesser extent is dLibra, Equella, Fedora, and MyCore. However, in a significant number of ROERs (14.5%), the platform was not identified, as it was not mentioned in their policies or anywhere else. This result coincides with that of OpenDOAR and Bueno-de-la-Fuente (2009), although recent studies (Amiel & Soares, 2016) have indicated a tendency to adopt CMS in the ROER scene.
Table 9
Software Platform
Metadata standards. The indexing of resources using metadata facilitates their location, dissemination and harvesting from other online platforms. In this analysis, the most widely used metadata standard in ROER was Dublin Core (DC, 50.9%). It was not specified whether simple or qualified DC was used, but both types may be involved, since additional qualifiers that further refine the meaning of OER have been found. The predominance of the DC element set might be due to the fact that DSpace uses it as its base metadata schema and DSpace was the most popular software used in ROER, as shown above.
However, the DC schema cannot efficiently characterize educational material or meet the needs of ROER (Koutsomitropoulos, Alexopoulos, Solomou, & Papatheodorou, 2010). Accordingly, several educational metadata standards have appeared over time, to improve the description of educational resources by identifying their pedagogical properties. A widely adopted standard is IEEE LOM (Learning Object Metadata), which has an inherent extension capability (Al-Khalifa and Davis, 2006) and attributes that enable learning resources to be managed and evaluated (Nair & Jeevan, 2004). Other learning specifications are IMS, which is a development of LOM, and SCORM (Sharable Content Object Reference Model) for sequencing structured educational material in the form of reusable content.
Table 10
Metadata Standards
Twenty per cent of the ROER adopted specific educational medatada standards for describing the OER (table 11). The most commonly used standards were LOM/SupLOMFR (14.5%), SCORM (0.91%), IMS (0.91%) and others (3.6%) (see Tables 10 and 11). Some other ROER provided a free tag system (9.1%). Lastly, 2.7% used a generic schema for describing digital objects, called METS/PLMET.
Table 11
Educational Metadata Standard
The pattern shown by these data contrasts with the findings of other studies, in which the importance of IEE LOM (29% of ROER) outstripped that of DC (22%) (Tzikopoulos et al., 2007) and in which less volume and fewer details of the metadata standard (37.7% included either DC or LOM) were identified (Atenas & Havemann, 2013).
Intentionality. A total of 36.3% of ROER expressed an intention to reuse materials, in the form of
Table 12
Intention to Reuse
Versioning. Thirteen of the ROER used some technical implementations to record different versions of the same resource. For instance, Jorum allows "multiple versions of an object to be deposited by adding information on the version in the description." Other ROER provided the following.
Of the subtotal of ROER that described reuse intentions (40 out of 110 repositories) (Table 13), only 32.5% (13 out of 40 ROER) included versioning. Therefore, a more intentional rather than a technical solution to support reuse has been observed. This may be because maintenance and updating are problems that ROER are struggling with (McGreal, 2011).
Table 13
Versioning
Licenses. Table 14 shows that only 5.5% of repositories had restricted access to resources (copyright with all rights reserved or login requirements). A total of 56.4% had an open license, 24.5 had both open and restricted licenses and, finally, 13.6% did not specify the license.
Amongst the open licenses, Creative Commons (CC) prevailed (43.6%), followed by specific licenses for educational use (10%), and other in-house licenses provided by the repositories (2.7%). As for ROER with a combination of both open and restricted access, 11 out of 27 were found to have CC.
Table 14
Licenses
In some studies, 76% of ROER had copyright policies (Tzikopoulos et al., 2007) and 52.5% (Atenas & Havemann, 2013) or 22% (Amiel & Soares, 2016) had CC licenses. Our results were more positive, since they showed an increase in openness of ROER and in the use of CC. However, there were still very few CC licenses, considering that they are relatively easy to adopt and lead to greater access, use and reuse of resources.
Our findings also revealed that CC are more commonly implemented in the USA and Europe, where more ROER are based. The opposite situation was found by Atenas & Havemann (2013), where the incidence of CC was greater in areas with fewer ROER (in Oceania, the Middle East and Africa).
The specific type of CC license provided by ROER can reveal the real opportunities they provide to reuse resources. Out of a total of 59 ROER with CC (Table 14), we observed (Table 15) that most allowed reuse (86.4%): 42.4% were Attribution-Noncommercial-Share Alike (BY-NC-SA), 27.1% used any of the six CC licenses, and 16.9% were under BY-NC. This last license allows others to non-commercially remix the original work, and, if it is also combined with SA (BY-NC-SA), new creations that build on the original work must be licensed under identical terms. The remaining 13.6% that did not allow any reuse corresponded to the most restrictive CC (BY-NC-ND), which only permits others to download and share resources if they credit the author, and does not allow the material to be changed in any way or used commercially.
Table 15
Creative Commons Licenses
The preponderance of the CC-BY-NC-SA license is in line with the findings of other studies (Venturini, 2014; Amiel & Soares, 2016), but was clearly higher in the current research (42.4% versus 8% found by Amiel).
Although the number of CC licenses that facilitated reuse was high (51 out of 59), it still only represented less than half of the total ROER (51 out of 110).
Granularity. It was found (Table 16) that just 20.9% of ROER had resources with different levels of granularity. Other research has shown that granular resources are more valued and effective for users (Wharrad & Windle, 2010) and play an important role in reuse for teaching and learning (Yalcinalp & Emiroglu, 2012).
Learning Space specifies that resources can be catalogued both as single entities (an entire manual) and constituent parts (chapters). Temoa also refers to resources as topics, activities or courses. ATE Central provides separate categories for "instructional unit" and "courses;" RUA provides specific metadata (c.relation.ispartof) to indicate that modules are part of a subject or course, and RADAR links the different units related to a course.
Table 16
Granularity
Granularity and the rest of the indicators of reuse (Table 17) were found to be interconnected. Thus, ROER with granularity also included social networks (78.3%), a quality system (73.9%) and some intention for reuse or versioning (60.9%).
Table 17
Indicators of Reuse in ROER with Granularity
Open formats. A total of 15.4% of the ROER included open formats (Table 18). Amongst these, only 9.1% implemented open formats as metadata; the other 6.3% included open formats, but they did not appear in the results.
Table 18
Open Formats
Ten of the 17 ROER with open formats also supported reuse (intentionally or through versioning), and 14 had open licenses (most of them were CC; some had licenses for educational use). These results are fairly consistent if we consider that open formats should facilitate reuse.
Some good practices regarding open formats in ROER were the inclusion of recommendations about their adoption, which were found in some guidelines: "attention should be given to the adoption of open standards. We would generally recommend resource authors to use open source solutions in preference to proprietary ones" (ReStore); "open formats ensure the preservation of resources" (Eprints UCM) and "whenever is possible, the open and easily re-usable formats should be used" (Oregon's repository). Finally, MERLOT is committed to creating materials with Content Builder and Pachyderm open software.
Some preferred formats that were detected are:.odt or.txt for text and.flac for audio (Oregon's repository), epub for ebooks (Open Learn), html/xml (DLynx, UVED, OpenStax CNX and UOH) and Css for web (Jorum).
Quality. A total of 43.6% of the ROER specifically mentioned quality, from a top-down perspective and according to three types of criteria (Table 19). This result contrasts with that of other studies (Castaño et al., 2016), in which quality assurance was even less present (11%).
Table 19
Top-Down Quality Criteria
According to our data, the most common approach when there is a form of quality assurance is internal assessment (20%), followed by the prestige of authors (13.6%) and external reviews (10%). This pattern is aligned with UNESCO (2011), which states that institutions are responsible for assuring the quality of OER used in teaching and learning environments, and should invest in improving quality before making materials available in ROER. The peer review, which is used extensively in scholarly publishing, has also become a quality assurance measure in ROER (Clements et al., 2015).
Institutions often ensure quality through the OER submission process (publishing rules) rather than the content. However, quality parameters exist in some cases, for example in the JISC Enriching Digital Resources Programme (in the First World War Poetry Digital Archive). Other ROER use guidelines that guarantee relevant OER: those with "learning objectives, instructional strategies and alignment with educational standards" (Exploratorium's Howtosmile), and with "an eminent educational or research purposes in teaching" (Open SNH). In other cases, a review checklist is provided before resources are uploaded (Humbox).
The main reasons given by ROER that provide quality through recognition of the author are that the authenticity of the content is the sole responsibility of the creator, and authors have been full-time lecturers for other organizations and are experts in their fields. With these arguments, ROER exempt themselves from any responsibility for content, as authors must meet certain conditions before submitting OER (Temoa) or must participate in all the development stages (Chem Collective).
Some ROER based on peer review systems rely on specialist groups (Learning Exchange) or editorial boards (Merlot) that decide which resources are suitable for their communities. Others make public their review criteria, such as "content quality, scope, beneficiaries, usability and potential ongoing value of the site" (Restore) and "high scientific accuracy, good pedagogical effectiveness, ease of use, clarity and completeness of documentation, motivating for learners, show robustness, and illustrate significance of content" (DLESE).
The few ROERs (3.6%) that do not explicitly assume liability for their resources (Table 19) are those stored in external containers (such as WLP), or not subject to quality control (DuEPublico).
A bottom-up quality approach based on user-generated evaluation (by rating and comments) will be analyzed in the next section.
Community of users. A total of 62.7% of ROER (69 out of 110) provided some kind of community engagement (in the remaining cases, this was an unknown factor). The most common interaction allowed was recommending OER by sharing them through social networks, electronic mail or RSS. This was found in approximately half of the ROER (54 out of 110). Conversely, less than one-quarter of ROER allowed comments (19 out of 110) or ratings (12 out of 110). Thus, in terms of ensuring a bottom-up quality system based on user action, ROER took less account of evaluation tools, such as rating, than of social media services.
Table 20
Community of Users (Bottom-Up Quality Criteria)
These findings are similar to some previous results (Atenas & Haveman, 2013), in which 51.2% of ROER allowed sharing, and 32.5% allowed ratings. However, it is difficult to discern what these numerical scores mean: are they positive enough to claim real community engagement or do they demonstrate the existence of review systems for users? Some authors (Zervas et al., 2014) found that social functionalities were not strong enough to assure quality (Ochoa & Duval, 2009). Clement (2016) added that they were insufficient if unsupported by a users' community. At the moment, these results show the extent to which community systems are available in ROER.
In a more qualitative analysis, some examples of ROER point to the existence of community spaces and behaviours. These include: discussion areas and special interest groups (in Promethean Plane t); groups of contributors that use and comment on OER (in Humboldt Digital Scholar and LORO); communities of interest and disciplines with their own YouTube channel and blogs for registered users (in MERLOT); and international communities involved in creating and adapting OER and participating in a blog and wiki (in Open Michigan). Other ROER have recognition systems to evaluate and receive recognition from the community. For example, some assess members' level of contribution (TEMOA) or provide "top members" ratings (Promethean Plane t).
Finally, a comparison of all the proposed indicators of reuse (Table 21) shows that while they do not have enough presence in ROER (all figures were below 50%), except licensing and social networks (close to or over 50%), most have a greater incidence in OER-exclusive repositories. In these, the features that were found most frequently were again licensing and social networks (with an increment of 20.9 points), as well as quality, intention and versioning. Note that although the figure for granularity is still low (38.3%), it is much higher than in the total ROER (17.4 points higher).
Table 21
Indicators of Reuse
OER types. The ROER (85.4%) provided information about the type of stored resources (Table 22). This information was usually found in the metadata "dc.format" or "dc.format.mimetype," other related metadata, or the search type drop-down list.
Table 22
Metadata on Type of Resources
Table 23 shows the wide range of resources that were identified in the ROER.
Table 23
Types of OERS
The content policies of the ROER tended to state what kind or resources could be included in the collections and who could include material. For instance, RODERIC asserted that "by teaching documents we mean any original material that may be of interest for use in the classroom" and LEEDSMET added that resources "may only be deposited by accredited members of the institution or their delegated agents". In addition, Open Access policies establish that doctoral theses, final degree projects and dissertations must be deposited (PADEMOS and RIUMA) and, in some cases educational resources must also be deposited (UCREA).
Some ROER provided resources specifically about teaching and innovation in teaching as disciplines (DUGI Docs) or offered specific collections on Curricula and Pedagogy (Belarusian National). Finally, PADEMOS provided a collection of "Material accessible to people with disabilities."
Educational metadata. A total of 42.7% of the ROER included specific metadata related to education (Table 24). The most popular educational metadata were "educational level or grade" (14.5%) and "intended or suggested use" (10.9%). Others were related to the intended audience, pedagogy, knowledge area, learning goals, learning time and related syllabus. Finally, in some cases in which there were no specific metadata for educational aspects, other information about targeted users or educational intention was located in fields designed for descriptions or comments.
Table 24
Types of Educational Metadata
Regarding the type of standards used by ROER with and without educational metadata (Table 25), most of the ROER that used DC did not include specific educational metadata. In contrast, the vast majority of ROER that used educational metadata standards based on LOM or SCORM (19 out of 22) also provided metadata on specific educational aspects. This pattern could be seen as a logical consequence of the inherent characteristics of each metadata standard. Therefore, a greater presence of educational standards is expected in ROER that provide educational data, since these standards have been created with this purpose.
Table 25
Educational Standards and Metadata in ROER
The results also showed another striking case: ROER with educational metadata using specific educational standards were equal in number to those using DC (18 in each case). This parity might be due to the possibility of mixing different metadata standards in the same repository, which is known as crosswalks metadata. This creates compatibility between descriptions, builds rich contexts for learning, and is useful for unnamed information that needs to be integrated (Godby, Young, & Childress, 2004). For instance, LOM elements can be incorporated in the default DSpace's qualified DC schema by direct mapping or by creating a further implementation (Koutsomitropoulos et al., 2010; Alexopoulos, Solomou, Koutsomitropoulos, & Papatheodorou, 2010; Skourlas et al., 2016). Therefore, DC metadata can be extended and optimized for education by incorporating educational descriptors. Some examples of metadata co-existing on the same platform can be seen in Jorum, which combines DC (dc.audience), LOM (lom.educational.context) and JMD packages (jmd.oer) in the same register, and ARES UNASUS, which has its own metadata system based on LOM (f.i, unasus.typicalLearningTime).
Learning goals. Just 29.1% of ROER included learning goals (Table 26). They were most commonly located in a description field (40.6%), but they were also found in specific descriptors for "Learning objectives" (37.5%) and "Instruction method/pedagogy" (9.3%) or included as part of the "Intended use" (3.1%) (see Table 27).
Table 26
Learning Goals
Table 27
Location of Learning Goals
Most of the ROER that excluded metadata on learning goals (55 out of 78) did not contain any other specific metadata on education. In contrast, ROER with learning goals as a descriptor also included other educational metadata (in 10 out 12) and used educational metadata standards (in 8 out of 12).
Finally, by crossing all the proposed educational indicators (Table 28), educational aspects were found to have a higher incidence (with notable increases in educational metadata, which were present in over 60%, and in learning goals in OER-exclusive repositories and a lower presence (all below 50%, except the type of OER) with respect to the total ROER.
Table 28
Indicators of Educational Aspects
Using the incidence of indicators of reuse and educational issues in ROER, we could identify the level of development of these repositories and establish four main types (Table 29).
Forty-four ROER (40% of the total) were placed in the top, high and medium categories, with different grades of promotion of reuse and educational features. However, the remaining 66 ROER, which make up the bulk of the population (60%), do not have enough drivers to facilitate reuse and educational issues.
Table 29
Categorization of ROER According to the Incidence of Reuse and Educational Indicators
An initial table was used to classify ROER according to the incidence of reuse and educational indicators. It contained the population of 110 ROER in the X-axis, and the 10 assessment indicators in the Y-axis: 6 on reuse (intentionality & versioning, open licenses or CC, granularity, open formats, quality and social networks) and 4 on educational aspects (OER type, educational standard, educational metadata and learning goals). A second table, presented above, was drawn up to cluster similar ROER (those which included more or fewer indicators) and to identify the most popular indicators and the correlations between them. This is a basic, exploratory categorization that will be analyzed in greater depth in future research.
Based on the analysis, it has been found that most ROER on higher education are institutional (mainly created by universities and government bodies) and exclusively designed for educational resources, rather than hybrids that also contain research content. Considering that OER production remains essentially linked to public funding, international projects and institutions (Castaño et al., 2016), it makes sense that OER production has no place outside of institutional funding. This is particularly true when there are few rewards for OER
The main current features identified in ROER for promoting reuse are open licenses (specifically CC) and social networks. Intentionality, versioning and quality lag behind, and granularity and open formats are less evident. This suggests that repositories focus more specifically on OA licensing for OER and on taking care of or facilitating the creation of communities of users, who, in turn, could offer additional bottom-up quality criteria for deposited content. The reuse promotion pattern was slightly different in ROER created specifically to deposit OER. The open licensing indicator remained important, and the social network aspect increased considerably. In addition, quality and granularity began to play a more prominent role. Therefore, OER-exclusive repositories include a wider range of features, and use them more intensively, to facilitate the reuse of their resources.
The most prevalent educational aspects offered in ROER are content type, format and subject. However, less than half of ROER provide specific metadata related to education, and just over one quarter include learning goals. In addition, the few ROER that provide educational metadata do not always use a specific educational standard such as LOM or SCORM, which were developed to better identify the pedagogical properties of OERs. However, some do incorporate LOM elements in other default metadata specifications, such as Dublin Core. This could be because many institutional repositories use DC as a general schema for describing their resources, since they were created to store not only educational material, but also research or other types of content.
We could assume that educational features are more present in OER-exclusive repositories, which are created to meet an educational need. However, such repositories are not currently achieving their fullest potential. Although there is more educational information in this kind of repository (just over sixty per cent), there are still many cases of OER described and retrieved by type or format instead of by detailed educational metadata that better meets the users' needs, for example the suggested use of content, its duration, pedagogy, target audience or learning goals.
To sum up, current ROERs include more drivers that promote the reuse of OERs, mainly through open licenses and social networks, than features facilitating the retrieval and use of OERs according to educational needs, such as learning goals.
A level of development of ROER was identified, based on the incidence of reuse and educational indicators. We identified four clusters of ROERs ranging from a small top level of repositories that include most of the reuse and educational dimensions to a large lower level of repositories that lack important features such as open formats, granularity and educational metadata. In the middle of this scale, we also identified a high category mainly characterized by providing social networks and educational metadata standards, and a medium category offering a diverse mix of indicators.
We summarize here some of our findings in relation to previous studies. Some similar trends were identified in the general description of the ROER: most of them were multidisciplinary, institutional, predominantly based in Europe and the USA, and supported by DSpace technology. However, there are also a few fine distinctions, in three particular areas: in the numbers, insofar as our data show a significant increase in institutional ROER; in location, given that Europe and especially the UK, Spain and France took precedence over the USA in our study, while other related work found the opposite situation; and finally in technical infrastructure, where in line with other recent studies and in contrast to previous studies, our findings show an emerging tendency to adopt CMS.
Some differences were also observed with respect to prior works. While our results showed a clear predominance of Dublin Core above other specifically educational metadata specifications, other studies found an opposite pattern or simply did not specify the metadata schema used by the ROER. Other differences observed in our research were increased openness in ROER and a more intensive use of CC licenses overall in areas with higher numbers of ROER, such as Europe and the USA.
Finally, some of our main contributions were that OER-exclusive repositories led to better reuse and educational outcomes than hybrids and that most of the CC licenses used in ROER allowed reuse, as there was a notable presence of BY-NC-SA and BY-NC. This achievement, added to the use of other kinds of open license, shows increased openness of ROERs.
To further develop the research, the evaluation framework could be expanded by adding features, and the categorization of ROER could be examined in greater depth. The research could use indicators that are more focused on pedagogy and bottom-up approaches, as these increase the educational usefulness of ROER and the reuse of OER. Additional features could be based on users' interaction, meaning communities and personal spaces, kind of information shared, social tagging procedures, etc., and types of OER, defined by whether the material was produced by students or educators and who it is aimed at (students, teacher trainers, the OER community, learning innovators). Users' participation helps to evaluate and license the stored OER, which facilitates reuse and makes it easier to integrate educational needs into the classification and retrieval process.
This study was carried as part of the project "Open Access to Science in Spain" (CSO2014-52830-P) of the Spanish R&D Plan, funded by the Spanish Ministry of Science and Innovation.
Alexopoulos, A. D., Solomou, G. D., Koutsomitropoulos, D. A., & Papatheodorou, T. S. (2010). Enhancing digital repositories with learning object metadata. In Lazarinis, Green, & Pearson (Eds.), Handbook of research on e-Learning standards and interoperability: frameworks and issues (pp.246-263). doi: 10.4018/978-1-61692-789-9
Al-Khalifa, H., & Davis, H. (2006). The evolution of metadata from standards to semantics in E-learning applications. In Proceedings of the Seventeenth Conference on Hypertext and Hypermedia, (pp. 69-72). New York: ACM Press. doi: 10.1145/1149941.1149956
Allen, N., Browne, D., Forward. M. L., Green, C., & Tarkowski, A. (2015). Foundation for OER strategy development (Electronic document). Retrieved from http://www.oerstrategy.org/home/read-the-doc/
Amiel, T., & Soares, T. (2016). Identifying tensions in the use of open licenses in OER repositories. The International Review of Research in Open and Distributed Learning, 17(3). doi: http://dx.DOI.org/10.19173/irrodl.v17i3.2426
Atenas, J. & Havemann L. (2013). Quality assurance in the open: An evaluation of OER repositories. The International Journal for Innovation and Quality in Learning, 1(2), 22-34. Retrieved from http://eprints.soas.ac.uk/17347/1/30-288-1-PB.pdf
Barrueco, J.M., De Miguel, M., González, C & Rico-Castro, P. (2014). Guía para la evaluación de repositorios institucionales de investigación (Guide for the evaluation of institutional research repositories). Madrid. Retrieved from http://hdl.handle.net/10272/11370
Bissell, A. (2011). OER and Open License: the dual-pub solution. The Monterey Institute for Technology and Education. Retrieved from https://oerknowledgecloud.org/sites/oerknowledgecloud.org/files/59893035-OER-and-Open-Licenses-Dual-Pub-Solution.pdf
Bueno-de-la-Fuente, G., Hernández-Pérez, T., Rodríguez-Mateos, D., Méndez-Rodriguez E., & Martín-Galán B. (2009). Study on the use of metadata for digital learning objects in University institutional repositories (MODERI). Cataloging & Classification Quarterly, 47(3-4), 262-285. doi: 10.1080/01639370902737315.
Camilleri, A. F., Ehlers, D. U., & Pawlowski, J. (2014). State of the art review of quality issues related to Open Educational Resources (OER) (JRC Science for Policy Report). doi: http://dx.doi.org/10.2791/80171
Castaño, J., Punie, Y., Inamorato, A., Mitic, M., & Morais, R. (2016). How are higher education institutions dealing with openness? A survey of practices, beliefs and strategies in five European countries (JRC Science for Policy Report). doi: 10.2791/709253
Cervone, H. F. (2011). Digital learning object repositories. OCLC systems & services: International Digital Library Perspectives, 28(1), 14-16. doi: 10.1108/10650751211197031
Chatzinotas, S., & Sampson, D. (2005). Exploiting the learning object paradigm for supporting web-based learning communities. Proceedings of the 4th IASTED International Conference on Web-based Education (pp. 165-170), Switzerland: ACTA Press.
Clements, K. (2016). Why Open Educational Resources repositories fail? The contribution of quality approaches to the success of repositories. University of Jyväskylä. Retrieved from http://urn.fi/URN:ISBN:978-951-39-6588-4
Clements, K., Pawlowski, J., & Manouselis, N. (2015). Open Educational Resources repositories literature review—Towards a comprehensive quality approaches framework. Computers in Human Behavior, 51, 1098-1106. doi: 10.1016/j.chb.2015.03.026
Cohen, A., Reisman, S., & Bied Sperling, B. (2015). Personal spaces in public repositories as a facilitator for Open Educational Resource usage. The International Review of Research in Open and Distributed Learning, 16(4). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2399
D'Antoni, S. (2007). Open Educational Resources and open content for higher education. RUSC. Universities and Knowledge Society Journal, 4(1). doi: https://doi.org/10.7238/rusc.v4i1.292
D'Antoni, S. (2013). A world map of Open Educational Resources initiatives: Can the global OER community design and build it together ? Summary report of an international conversation. OER KnowledgeCloud. Retrieved from https://oerknowledgecloud.org/?q=content/world-map-open-educational-resources-initiatives-can-global-oer-community-design-and-build-i
De los Arcos, B., Farrow, R., Perryman, L. A., Pitt, R., & Weller, M. (2014). OER evidence report 2013-2014. OER Research Hub. Retrieved from http://oro.open.ac.uk/41866/1/oerrh-evidence-report-2014.pdf
Dichev, C., & Dicheva, D. (2012). Is it time to change the OER repositories role? In Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2012). Washington, DC, USA.
European Commission. (2013a). Communication from the Commission to the European Parliament. Opening up education: Innovative teaching and learning for all through new technologies and Open Educational Resources. Retrieved from http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52013DC0654&from=EN
European Commission (2013b). Analysis and mapping of innovative teaching and learning for all through new technologies and Open Educational Resources in Europe. Retrieved from http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1389115521455&uri=CELEX:52013SC0341
European Commission (2014). Report on progress in quality assurance in higher education. Retrieved from http://ecahe.eu/assets/uploads/2014/01/EU-Report-on-Progress-in-Quality-Assurance-in-Higher-Education-2014.pdf
Fulantelli, G., Gentile, M., Taibi, D., & Allegra, M. (2008). The Open Learning Object model to promote Open Educational Resources. Journal of Interactive Media in Education, 2008(1). doi: http://DOI.org/10.5334/2008-9
Gesser, G. (Ed., 2007). Open educational practices and resources. OLCOS Roadmap 2012 (PDF document). Retrieved from http://www.olcos.org/cms/upload/docs/olcos_roadmap.pdf
Godby, C., Young, J., & Childress, E. (2004). A Repository of Metadata Crosswalks. DLib Magazine, 10(12). doi: 10.1045/december2004-godby.
Hylén, J., Damme, D. V., Mulder, F., & D'Antoni, S. (2012). Open Educational Resources: Analysis of responses to the OECD country questionnaire. OECD Education Working Papers, 76. doi: http://dx.DOI.org/10.1787/5k990rjhvtlv-en
Jacobi, R., & van der Woert, N. (2012, March). Trend report: Open Educational Resources 2012. Special Interest Report Open Educational Resources. Retrieved from https://www.surfspace.nl/media/bijlagen/artikel-697-ee18ac0f1441bb158e6122818f5f589e.pdf
Koutsomitropoulos, D. A., Alexopoulos, A. D., Solomou, G. D., & Papatheodorou, T. S. (2010). The use of metadata for educational resources in digital repositories: practices and perspectives. D-Lib Magazine, 16(1). doi: 10.1045/january2010-koutsomitropoulos
Krippendorff, K. (1980). Content analysis: An introduction to its methodology. London, Los Angeles: SAGE.
Lane, A., & McAndrew, P. (2010). Are open educational resources systematic or systemic change agents for teaching practice? British Journal of Educational Technology, 41(6), 952-962.
Littlejohn, A. (2003). Issues in reusing online resources. In A. Littejohn (Ed.). Reusing online resources: A sustainable approach to elearning (pp. 1-8). London: Kogan Page.
Maina, M.F., & Guàrdia, L. (2012). Creating OER for versioning: Experts and learners as knowledge contributors. In Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare (pp. 250-259). AACE. Retrieved from https://www.learntechlib.org/p/41596
McGreal, R. (Ed.). (2004). Online education using learning objects. New York: RoutledgeFalmer.
McGreal, R. (2011). Open Educational Resource repositories: An analysis. Athabasca University. Retrieved from http://www.academia.edu/2832823/Open_educational_resource_repositories_An_analysis
McKerlich, R. C., Ives C., & McGreal R. (2013). Measuring use and creation of open educational resources in higher education. The International Review of Research in Open and Distance Learning. 14(4), 91-102. Retrieved from https://oerknowledgecloud.org/content/measuring-use-and-creation-open-educational-resources-higher-education
Monge, S., Ovelar, R., & Azpeitia, I. (2008). Repository 2.0: Social dynamics to support community building in learning object repositories. Interdisciplinary Journal of E-Learning and Learning Object, 4, 191-204.
Nair, S., & Jeevan, V. (2004). A brief overview of metadata formats. DESIDOC Bulletin of Information Technology, 24(4), 3-11.
Neumann, J., & Muuß-Merholz, J. (2016). OER Atlas 2016 Open Educational Resources: Akteure und Aktivitäten in Deutschland, Österreich und der Schweiz (Electronic document). Retrieved from https://oerworldmap.files.wordpress.com/2016/03/oer-atlas-2016-komplett.pdf
Ochoa, X., & Duval, E. (2009). Automatic evaluation of metadata quality in digital repositories. International Journal on Digital Libraries, 10(67). doi: 10.1007/s00799-009-0054-4
Organisation for Economic Co-Operation and Development. (2007). Giving knowledge for free: The emergence of open educational resources. Centre for Educational Research and Innovation. Paris. Retrieved from https://www.oecd.org/edu/ceri/38654317.pdf
Orr, D., Rimini, M., & Van Damme, D. (2015). Open educational resources: A catalyst for innovation. Paris. doi: http://dx.doi.org/10.1787/9789264247543-en
Palavitsinis, N., Manouselis, N., & Sanchez-Alonso, S. (2014). Metadata quality in learning object repositories: A case study. The Electronic Library, 32(1), 62-82.
Pegler, C. (2011). Reuse and repurposing of online digital learning resources within UK higher education: 2003-2010 (Doctoral dissertation). The Open University. Retrieved from http://oro.open.ac.uk/32317
Rodríguez, J., Dodero J., & Sánchez-Alonso, S. (2011). Determinando la relevancia de los recursos educativos abiertos a través de la integración de diferentes indicadores de calidad (Determining the relevance of open educational resources through the integration of different quality indicators). Revista de Universidad y Sociedad del Conocimiento, 8(2), 46-60.
Sampson, D., Zervas, P., & Sotiriou, S. (2013). Sharing of open science education resources and educational practices in Europe. In R, McGreal, W. Kinuthia, & S. Marshall (Eds.). Open educational resources: Innovation, research and practice, (pp. 105-121). Retrieved from https://oerknowledgecloud.org/sites/oerknowledgecloud.org/files/pub_PS_OER-IRP_web.pdf#page=131
Santos-Hermosa, G., Ferran, N., & Abadal, E. OER repositories. (2015, June 4-5). Characterisation and evaluation indicators. Poster session presented at the meeting of II Internationale Seminar on LIS Education and Research (LIS-ER), Barcelona, Spain. Retrieved from http://bd.ub.edu/liser/sites/bd.ub.edu.liser/files/posters/Poster_LIS-ER_Santos%252c%20Ferran%26Abadal_def.pdf
Serrano, R., Melero, R., & Abadal, E. (2014). Indicadores para la evaluación de repositorios institucionales de acceso abierto [Indicators for an Evaluation of Institutional Repositories of Open Access]. Anales De Documentación, 17(2). doi: https://doi.org/10.6018/analesdoc.17.2.190821
Shmueli, E., & Cohen, A. (2012). The usage of open educational resources in MAOR repository. The International Journal of Technology Enhanced Learning (IJTEL), 4(3), 284-298. doi: 10.1504/IJTEL.2012.051552
Skourlas, C., Tsolakidis, A., Belsis, P., Vassis , D., Kampouraki , A., Kakoulidis, P., & Giannakopoulos, A. (2016). Integration of institutional repositories and e-learning platforms for supporting disabled students in the higher education context, Library Review, 65(3), 136-159.
Sicilia, M., García-Barriocanal, E., Sánchez-Alonso, S., & Cechinel, C. (2010). Exploring user-based recommender results in large learning object repositories: The case of MERLOT. Procedia Computer Science, 1, 2859-2864.
Tzikopoulos, A., Manouselis, N., & Vuorikari, R. (2007). An overview of learning object repositories. In P. Taylor Northrup (Ed.), Learning objects for instruction: Design and evaluation (pp. 29-55). IGI Global. doi: 10.4018/978-1-59904-334-0.ch003
UNESCO. (2011). Guidelines for open educational resources (OER) in higher education. Retrieved from http://unesdoc.unesco.org/images/0021/002136/213605e.pdf
Venturini, J. (2014). Recursos educacionais abertos no Brasil: O campo, os recursos e sua apropriação em sala de aula (Open Educational Resources in Brasil: The field, the resources and its appropriation in the classroom). Questão, 11. Retrieved from http://www.acaoeducativa.org/images/stories/pdfs/EmQuestao11_site.pdf
Westell, M. (2006). Institutional repositories: Proposed indicators of success. Library Hi Tech, 24(2), 211-226. http://www.acaoeducativa.org/images/stories/pdfs/EmQuestao11_site.pdf
Wharrad, H., & Windle, R. (2010). Case studies of creating reusable inter professional e-learning objects. In A. Bromage, L. Clouder, J. Thistlethwaite, & F. Gordon (Eds.). Interprofessional e-learning and collaborative work: practices and technologies. IGI Global (pp. 260-274).
Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy.In D. A. Wiley (Ed.). The instructional use of learning objects. Open Publication License. Retrieved from http://reusability.org/read/chapters/wiley.doc
Wiley, D., Bliss, T. J., & McEwen, M. (2014). Open educational resources: A review of the literature. In J. M. Spector, M. D. Merrill, J. Elen & M. J. Bishop (Eds.). Handbook of research on educational communications and technology (pp. 781-789). New York: Springer. doi: 10.1007/978-1-4614-3185-5_63
Wills, S., & Pegler, C. (2016). A deeper understanding of reuse: learning designs, activities, resources and their contexts. Journal of Interactive Media in Education. 2016(1). doi: http://DOI.org/10.5334/jime.405
Windle, R. J., Wharrad, H., McCormick, D., Laverty, H., & Taylor, M. G (2010). Sharing and reuse in OER: experiences gained from open reusable learning objects in health. Journal of Interactive Media in Education, 2010(1), 4. doi: http://DOI.org/10.5334/2010-4
Yalcinalp S., & Emiroğlu, B. G. (2012). Through efficient use of LORs: Prospective teachers' views on operational aspects of learning object repositories. British Journal of Educational Technology, 43(3), 474-484. doi: 10.1111/j.1467-8535.2011.01212.x
Zancanaro, A., Todesco, J., & Ramos, F. (2015). A bibliometric mapping of open educational resources. The International Review of Research in Open and Distributed Learning, 16(1). doi: http://dx.DOI.org/10.19173/irrodl.v16i1.1960
Zervas, P., Alifragkis, C. & Sampson, D. G. (2014). A quantitative analysis of learning object repositories as knowledge management systems. Knowledge Management & E-Learning, 6(2), 156-170.
Repositories of Open Educational Resources: An Assessment of Reuse and Educational Aspects by Gema Santos-Hermosa, Núria Ferran-Ferrer, and Ernest Abadal is licensed under a Creative Commons Attribution 4.0 International License.