Repositories of Open Educational Resources: An Assessment of Reuse and Educational Aspects

This article provides an overview of the current state of repositories of open educational resources (ROER) in higher education at international level. It analyses a series of educational indicators to determine whether ROER can meet the specific needs of the education context, and to clarify understanding of the reuse of open educational resources (OER) provided by ROER. The aim of the study is to assess ROER by combining these two perspectives, and to form a basis for discussion among the universities that are responsible for these repositories. The method was based on content analysis and consisted of two phases: an exploration of international sources, and an analysis of 110 ROER using the proposed set of indicators. The results focus on data from the analysis of ROER websites and some models of good practices. They are presented according to three core dimensions for evaluating ROER: general factors to establish types of ROER, a focus on drivers for OER reuse, and a focus on educational aspects. It was found that most of the ROER that included one or more of the proposed reuse indicators were created exclusively for educational resources. Educational aspects are not yet firmly embedded into ROER. The few repositories that seem to have successfully included them are those that provide other educational metadata and use educational standards.


Article abstract
This article provides an overview of the current state of repositories of open educational resources (ROER) in higher education at international level. It analyses a series of educational indicators to determine whether ROER can meet the specific needs of the education context, and to clarify understanding of the reuse of open educational resources (OER) provided by ROER. The aim of the study is to assess ROER by combining these two perspectives, and to form a basis for discussion among the universities that are responsible for these repositories.
The method was based on content analysis and consisted of two phases: an exploration of international sources, and an analysis of 110 ROER using the proposed set of indicators. The results focus on data from the analysis of ROER websites and some models of good practices. They are presented according to three core dimensions for evaluating ROER: general factors to establish types of ROER, a focus on drivers for OER reuse, and a focus on educational aspects. It was found that most of the ROER that included one or more of the proposed reuse indicators were created exclusively for ed ucational resources. Educational aspects are not yet firmly embedded into ROER. The few repositories that seem to have successfully included them are those that provide other educational metadata and use educational standards.

Introduction
In the past decade, the Open Educational Resources (OER) movement has gained ground and expanded exponentially. Previously, there were only localized OER initiatives; now OER are recognized and

Background
Global knowledge and research in OER is increasing. This topic has aroused the interest of researchers, as OER could make a strong contribution to improving education and fostering openness.
However, none of these studies specifically refers to the reuse and educational aspects of ROER: these two unresolved issues merit particular attention in our research. Wiley (2000) considered that any digital resource could be reused to support learning, and subsequent studies have stressed the importance of reuse. However, they have also recognized the great difficulty in finding evidence of reuse (Orr, Rimini, & Van Damme, 2015), the still insufficient analyses (de los Arcos et al., 2014), and the "problem of remix" (Wiley, Bliss, & McEwen, 2014). The most common discussion framework has been focused on user surveys (McKerlich, Ives , & McGreal, 2013;Camilleri, Ehlers, & Pawlowski, 2014) and tracking OER usage (Pegler, 2011). A couple of studies specifically address the connection between reuse and ROER as a key theme that represents one type of activity within a repository (Atenas & Havemann, 2013), or link reuse to the creation of personal spaces (Cohen, Reisman, & Sperling, 2015). Nevertheless, ROER's current "predisposition" to reuse has not been considered. For this reason, the present research examines opportunities provided by ROER that facilitate reuse of OER.
We are aware of the connection between reuse and educational aspects, and understand that reuse can be more difficult because the educational design is rarely visible (Wiley et al., 2014). Therefore, this is the first review that integrates both perspectives in ROER. It also provides an overview of the current state of ROER in higher education.
As some aspects of ROER are underdeveloped, a distinctive, value-added framework is becoming indispensable. Since reuse and educational aspects in ROER still pose certain challenges, we propose the following research questions:

Method
The method used in this study was based on content analysis (Krippendorff, 1980) and consisted of two phases: 1. Exploration of international and specialized sources about ROER, and selection of the ROER population to be analyzed.
2. Analysis of the ROER population using a proposed set of indicators, divided into three main categories that were directly related to the research questions.

Selecting a Population
In order to select ROER for analysis, the following sources were used: the Registry of Open Access Repository (ROAR), the Directory of Open Access Repositories (Open DOAR), the portal Open Education Europe, a report on the State of the Art by the Educational Repositories Network (EdReNe), and other literature on the subject.
After retrieving a total of 1,186 repositories of "learning objects" or "educational resources," a population of 110 ROER was selected using the following criteria ( Figure 1): A. A higher education (HE) level of education, or HE combined with other levels.
B. Updated from 2011 onwards, and currently functioning.
C. Repositories with specific OER collections.
D. Repositories containing at least 50 OER.

Core Dimensions and Indicators
Repositories tend to be evaluated using institutional factors that are based on institutional needs and strategies (Westell, 2006;Serrano, Melero & Abadal, 2014). However, considering that ROER's particular requirements differ from those of traditional digital repositories (Cervone, 2011), other aspects could be taken into account.
The current study proposes three core dimensions for evaluating ROER: general/descriptive factors to establish types of ROER, a focus on drivers for OER reuse, and a focus on educational aspects. These categories are broken down according to specific indicators.

89
Category 1: General indicators: Types and technology.

ROER Types and Technology
Category 2: Indicators of reuse. One of the benefits of ROER is that teachers have the opportunity to "reuse, modify, and adapt resources" (Hylén, Damme, Mulder, & D'Antoni, 2012), and a secure and reusable platform is an important design factor that contributes to reuse (Wills & Pegler, 2016). Hence, ROER that implement drivers contributing to reuse would become "beneficial reusable platforms" for users. Table 2 shows a proposal of facilitating factors, and gives reasons for this chosen focus. Table 2 Indicators for Reuse Category 3: Educational indicators. Since it has been emphasized that the pedagogical value of educational resources is the most important factor (McGreal,200 4), Table 3 proposes a series of indicators that are designed to describe ROER from an educational perspective. Table 3 Educational Indicators

Data Gathering and Analysis
Content analysis was carried out for each of the 110 repository websites, using Tables 1 to 3 as codification frameworks. Tools for retrieving information from each website were used, such as searching (filters and metadata) and browsing (documentation). Data were collected between 2015 and 2016.
Content analysis limitations were broken URLs, duplicate data and the omission of pertinent data. Another limitation of this study was that some repositories are available in other languages, and have not been translated into English.

Results and Discussion
This section presents and discusses data from the analysis of ROER websites. In addition, some relevant ROER are highlighted as models of good practices.

ROER Types and Technology
Main features. Table 4 indicates that there are more OER repositories that only contain learning resources (54.5%) than hybrid repositories (45.5%) containing research and learning resources. Table 4 OER Stored Wholly or Partially Most ROER (Tables 5 and 6) were multidisciplinary (73.6%). The second most common discipline in ROER was Science, Technology, Engineering, and Maths (STEM, 12.7%). Most ROER were dedicated to higher education (85.5%); a lower number covered different levels of education (14.5%). Although a multidisciplinary approach also seemed more common in a previous analysis (OECD, 2007), other data suggest that single-subject repositories would be more suitable for teachers, since it is harder for teachers to find and use relevant OER for their subjects in broader repositories (Sampson, Zervas, & Sotiriou, 2013). Table 5 Disciplines Table 6 Educational Level 106 Authorship. It was found that the prevailing ROER were institutional (76.4% created by universities, foundations or institutes) and the rest were national (23.6% funded by governments, national organizations or consortia). This pattern is similar to that found in another analysis (Atenas & Havemann, 2013), in which institutional ROER were also predominant, but to a lesser extent (50% of the total, while the rest was broken down into different types).
The success of ROER in the UK might be due to the considerable funding provided for OER projects by the Joint Information Systems Committee (JISC). This pattern varies in other studies (Bueno -de-la-Fuente et al., 2009), in which the USA was found to be the most common location (28%), followed by Germany and Italy. However, our results might be influenced by the fact that they only focus on higher education ROER. The countries that were found to be most involved in ROER may coincide with the countries in which most publications about OER have been identified (Zancanaro et al., 2015). Table 7 Geographical Origin of ROER (Continents) Table 8 Geographical Origin of ROER (Countries) Software platform. Data on the technology that supports ROERs platforms (Table 9) suggests that DSpace is the most popular software (34.5%), followed by a range of other software (19.1%), some content management systems (CMS) and learning management systems (LMS) (11.8%), and Eprints (7.3%). Other software that is used to a lesser extent is dLibra, Equella, Fedora, and MyCore. However, in a significant number of ROERs (14.5%), the platform was not identified, as it was not mentioned in their policies or anywhere else. This result coincides with that of OpenDOAR and Bueno-de-la-Fuente (2009), although recent studies (Amiel & Soares, 2016) have indicated a tendency to adopt CMS in the ROER scene. Table 9 Software Platform Metadata standards. The indexing of resources using metadata facilitates their location, dissemination and harvesting from other online platforms. In this analysis, the most widely used metadata standard in ROER was Dublin Core (DC, 50.9%). It was not specified whether simple or qualified DC was used, but both types may be involved, since additional qualifiers that further refine the meaning of OER have been found. The predominance of the DC element set might be due to the fact that DSpace uses it as its base metadata schema and DSpace was the most popular software used in ROER, as shown above.
However, the DC schema cannot efficiently characterize educational material or meet the needs of ROER . Accordingly, several educational metadata standards have appeared over time, to improve the description of educational resources by identifying their pedagogical properties. A widely adopted standard is IEEE LOM (Learning Object Metadata), which has an inherent extension capability (Al-Khalifa and Davis, 2006) and attributes that enable learning resources to be managed and evaluated (Nair & Jeevan, 2004). Other learning specifications are IMS, which is a development of LOM, and SCORM (Sharable Content Object Reference Model) for sequencing structured educational material in the form of reusable content. Table 10 Metadata Standards Twenty per cent of the ROER adopted specific educational medatada standards for describing the OER (table 11). The most commonly used standards were LOM/SupLOMFR (14.5%), SCORM (0.91%), IMS (0.91%) and others (3.6%) (see Tables 10 and 11). Some other ROER provided a free tag system (9.1%).
Lastly, 2.7% used a generic schema for describing digital objects, called METS/PLMET.

Educational Metadata Standard
The pattern shown by these data contrasts with the findings of other studies, in which the importance of IEE LOM (29% of ROER) outstripped that of DC (22%) (Tzikopoulos et al., 2007) and in which less volume and fewer details of the metadata standard (37.7% included either DC or LOM) were identified (Atenas & Havemann, 2013).

Reuse in ROER
Intentionality. A total of 36.3% of ROER expressed an intention to reuse materials, in the form of  comments in the mission or in the collection's policies, which were found in ARIADNE, the University of Leicester's OER Repository or Xpert, for example;  advice and instructions on the reuse of resources. Open Michigan has a "Use Open Content" section, with recommendations on how to adapt and license resources; and  messages encouraging users to remix resources. NCLOR proposes that teachers can become more productive and save valuable time if they contribute new versions of resources. OER Africa also encourages teachers to adapt materials to African contexts and learners.

Table 12
Intention to Reuse Versioning. Thirteen of t he ROER used some technical implementations to r e c o r d different versions of the same resource. For instance, Jorum allows "multiple versions of an object to be deposited by adding information on the version in the description." Other ROER provided the following.
 Search filters to retrieve available versions, including information on where a specific resource was used (RADAR), how to reuse resources (HUMBOX), and where to find resources to remix or make derivatives (OER Commons).
 The most developed ROER showed chronologically who, when, and which files had been changed (DuEPublico); offered assistance about how to reuse or create contents that are suitable for reuse (in TEMOA "you can mix and match other people's items to build new ones and adapt them to your needs") and recommended how to reuse and attribute educational content (OpenStax CNX).
Of the subtotal of ROER that described reuse intentions (40 out of 110 repositories) (Table 13), only 32.5% (13 out of 40 ROER) included versioning. Therefore, a more intentional rather than a technical solution to support reuse has been observed. This may be because maintenance and updating are problems that ROER are struggling with (McGreal, 2011). Table 13 Versioning Licenses. Amongst the open licenses, Creative Commons (CC) prevailed (43.6%), followed by specific licenses for educational use (10%), and other in-house licenses provided by the repositories (2.7%). As for ROER with a combination of both open and restricted access, 11 out of 27 were found to have CC.

Table 14
Licenses In some studies, 76% of ROER had copyright policies (Tzikopoulos et al., 2007) and 52.5% (Atenas & Havemann, 2013) or 22% (Amiel & Soares, 2016) had CC licenses. Our results were more positive, since they showed an increase in openness of ROER and in the use of CC. However, there were still very few CC licenses, considering that they are relatively easy to adopt and lead to greater access, use and reuse of resources.
Our findings also revealed that CC are more commonly implemented in the USA and Europe, where more ROER are based. The opposite situation was found by Atenas & Havemann (2013), where the incidence of CC was greater in areas with fewer ROER (in Oceania, the Middle East and Africa).
The specific type of CC license provided by ROER can reveal the real opportunities they provide to reuse resources. Out of a total of 59 ROER with CC (Table 14), we observed (Table 15) that most allowed reuse (86.4%): 42.4% were Attribution-Noncommercial-Share Alike (BY-NC-SA), 27.1% used any of the six CC licenses, and 16.9% were under BY-NC. This last license allows others to non-commercially remix the original work, and, if it is also combined with SA (BY-NC-SA), new creations that build on the original work must be licensed under identical terms. The remaining 13.6% that did not allow any reuse corresponded to the most restrictive CC (BY-NC-ND), which only permits others to download and share resources if they credit the author, and does not allow the material to be changed in any way or used commercially.

Table 15
Creative Commons Licenses The preponderance of the CC-BY-NC-SA license is in line with the findings of other studies (Venturini, 2014;Amiel & Soares, 2016), but was clearly higher in the current research (42.4% versus 8% found by Amiel).
Although the number of CC licenses that facilitated reuse was high (51 out of 59), it still only represented less than half of the total ROER (51 out of 110).
Granularity. It was found (Table 16) that just 20.9% of ROER had resources with different levels of granularity. Other research has shown that granular resources are more valued and effective for users  and play an important role in reuse for teaching and learning (Yalcinalp & Emiroglu, 2012).
Learning Space specifies that resources can be catalogued both as single entities (an entire manual) and constituent parts (chapters). Temoa also refers to resources as topics, activities or courses. ATE Central provides separate categories for '"instructional unit" and "courses;" RUA provides specific metadata (c.relation.ispartof) to indicate that modules are part of a subject or course, and RADAR links the different units related to a course.

Table 16
Granularity Granularity and the rest of the indicators of reuse (Table 17) were found to be interconnected. Thus, ROER with granularity also included social networks (78.3%), a quality system (73.9%) and some intention for reuse or versioning (60.9%). Table 17 Indicators of Reuse in ROER with Granularity Open formats. A total of 15.4% of the ROER included open formats (Table 18). Amongst these, only 9.1% implemented open formats as metadata; the other 6.3% included open formats, but they did not appear in the results. Table 18 Open Formats  Some preferred formats that were detected are: .odt or .txt for text and .flac for audio (Oregon's repository), epub for ebooks (Open Learn), html/xml (DLynx, UVED, OpenStax CNX and UOH) and Css for web (Jorum).

Quality.
A total of 43.6% of the ROER specifically mentioned quality, from a top-down perspective and according to three types of criteria (Table 19). This result contrasts with that of other studies (Castaño et al., 2016), in which quality assurance was even less present (11%).

Table 19
Top-Down Quality Criteria According to our data, the most common approach when there is a form of quality assurance is internal assessment (20%), followed by the prestige of authors (13.6%) and external reviews (10%). This pattern is aligned with UNESCO (2011), which states that institutions are responsible for assuring the quality of OER used in teaching and learning environments, and should invest in improving quality before making materials available in ROER. The peer review, which is used extensively in scholarly publishing, has also become a quality assurance measure in ROER (Clements et al., 2015).
Institutions often ensure quality through the OER submission process (publishing rules) rather than the content. However, quality parameters exist in some cases, for example in the JISC Enriching Digital Resources Programme (in the First World War Poetry Digital Archive). Other ROER use guidelines that guarantee relevant OER: those with "learning objectives, instructional strategies and alignment 115 with educational standards" (Exploratorium's Howtosmile), and with "an eminent educational or research purposes in teaching" (Open SNH). In other cases, a review checklist is provided before resources are uploaded (Humbox).
The main reasons given by ROER that provide quality through recognition of the author are that the authenticity of the content is the sole responsibility of the creator, and authors have been full-time lecturers for other organizations and are experts in their fields. With these arguments, ROER exempt themselves from any responsibility for content, as authors must meet certain conditions before submitting OER (Temoa) or must participate in all the development stages (Chem Collective).
Some ROER based on peer review systems rely on specialist groups (Learning Exchange) or editorial boards (Merlot) that decide which resources are suitable for their communities. Others make public their review criteria, such as "content quality, scope, beneficiaries, usability and potential ongoing value of the site" (Restore) and "high scientific accuracy, good pedagogical effectiveness, ease of use, clarity and completeness of documentation, motivating for learners, show robustness, and illustra te significance of content" (DLESE).
The few ROERs (3.6%) that do not explicitly assume liability for their resources (Table 19) are those stored in external containers (such as WLP), or not subject to quality control (DuEPublico).
A bottom-up quality approach based on user-generated evaluation (by rating and comments) will be analyzed in the next section.

Community of users.
A total of 62.7% of ROER (69 out of 110) provided some kind of community engagement (in the remaining cases, this was an unknown factor). The most common interaction allowed was recommending OER by sharing them through social networks, electronic mail or RSS. This was found in approximately half of the ROER (54 out of 110). Conversely, less than onequarter of ROER allowed comments (19 out of 110) or ratings (12 out of 110). Thus, in terms of ensuring a bottom-up quality system based on user action, ROER took less account of evaluation tools, such as rating, than of social media services.

Table 20
Community of Users (Bottom-Up Quality Criteria) These findings are similar to some previous results (Atenas & Haveman, 2013), in which 51.2% of ROER 116 allowed sharing, and 32.5% allowed ratings. However, it is difficult to discern what these numerical scores mean: are they positive enough to claim real community engagement or do they demonstrate the existence of review systems for users? Some authors (Zervas et al., 2014) found that social functionalities were not strong enough to assure quality (Ochoa & Duval, 2009). Clement (2016) added that they were insufficient if unsupported by a users' community. At the moment, these results show the extent to which community systems are available in ROER.
In a more qualitative analysis, some examples of ROER point to the existence of community spaces and behaviours. These include: discussion areas and special interest groups (in Promethean Planet); groups of contributors that use and comment on OER (in Humboldt Digital Scholar and LORO); communities of interest and disciplines with their own YouTube channel and blogs for registered users (in MERLOT); and international communities involved in creating and adapting OER and participating in a blog and wiki (in Open Michigan). Other ROER have recognition systems to evaluate and receive recognition from the community. For example, some assess members' level of contribution (TEMOA) or provide "top members" ratings (Promethean Planet).
Finally, a comparison of all the proposed indicators of reuse (Table 21) shows that while they do not have enough presence in ROER (all figures were below 50%), except licensing and social networks (close to or over 50%), most have a greater incidence in OER-exclusive repositories. In these, the features that were found most frequently were again licensing and social networks (with an increment of 20.9 points), as well as quality, intention and versioning. Note that although the figure for granularity is still low (38.3%), it is much higher than in the total ROER (17.4 points higher).

Table 21
Indicators of Reuse Educational Aspects OER types. The ROER (85.4%) provided information about the type of stored resources (Table  22). This information was usually found in the metadata "dc.format" or "dc.format.mimetype," other 117 related metadata, or the search type drop-down list. Table 22 Metadata on Type of Resources Table 23 shows the wide range of resources that were identified in the ROER.

Table 23
Types of OERS The content policies of the ROER tended to state what kind or resources could be included in the collections and who could include material. For instance, RODERIC asserted that "by teaching documents we mean any original material that may be of interest for use in the classroom" and LEEDSMET added that resources "may only be deposited by accredited members of the institution or their delegated agents". In addition, Open Access policies establish that doctoral theses, final degree projects and dissertations must be deposited (PADEMOS and RIUMA) and, in some cases educational resources must also be deposited (UCREA).
Some ROER provided resources specifically about teaching and innovation in teaching as disciplines (DUGI Docs) or offered specific collections on Curricula and Pedagogy (Belarusian National). Finally, PADEMOS provided a collection of "Material accessible to people with disabilities." Educational metadata. A total of 42.7% of the ROER included specific metadata related to education (Table 24). The most popular educational metadata were "educational level or grade" (14.5%) and "intended or suggested use" (10.9%). Others were related to the intended audience, pedagogy, knowledge area, learning goals, learning time and related syllabus. Finally, in some cases in which there were no specific metadata for educational aspects, other information about targeted users or educational intention was located in fields designed for descriptions or comments.

Types of Educational Metadata
Regarding the type of standards used by ROER with and without educational metadata (Table 25), most of the ROER that used DC did not include specific educational metadata. In contrast, the vast majority of ROER that used educational metadata standards based on LOM or SCORM (19 out of 22) also provided metadata on specific educational aspects. This pattern could be seen as a logical consequence of the inherent characteristics of each metadata standard. Therefore, a greater presence of educational standards is expected in ROER that provide educational data, since these standards have been created with this purpose.

Educational Standards and Metadata in ROER
The results also showed another striking case: ROER with educational metadata using specific educational standards were equal in number to those using DC (18 in each case). This parity might be due to the possibility of mixing different metadata standards in the same repository, which is known as crosswalks metadata. This creates compatibility between descriptions, builds rich contexts for learning, and is useful for unnamed information that needs to be integrated (Godby, Young, & Childress, 2004). For instance, LOM elements can be incorporated in the default DSpace's qualified DC schema by direct mapping or by creating a further implementation Skourlas et al., 2016). Therefore, DC metadata can be extended and optimized for education by incorporating educational descriptors. Some examples of metadata co-existing on the same platform can be seen in Jorum, which combines DC (dc.audience), LOM (lom.educational.context) and JMD packages (jmd.oer) in the same register, and ARES UNASUS, which has its own metadata system based on LOM (f.i, unasus.typicalLearningTime).
Learning goals. Just 29.1% of ROER included learning goals (Table 26). They were most commonly located in a description field (40.6%), but they were also found in specific descriptors for "Learning objectives" (37.5%) and "Instruction method/pedagogy" (9.3%) or included as part of the "Intended use" (3.1%) (see Table 27).

Table 26
Learning Goals Table 27 Location of Learning Goals Most of the ROER that excluded metadata on learning goals (55 out of 78) did not contain any other specific metadata on education. In contrast, ROER with learning goals as a descriptor also included other educational metadata (in 10 out 12) and used educational metadata standards (in 8 out of 12).
Finally, by crossing all the proposed educational indicators (Table 28), educational aspects were found to have a higher incidence (with notable increases in educational metadata, which were present in over 60%, and in learning goals in OER-exclusive repositories and a lower presence (all below 50% , except the type of OER) with respect to the total ROER.

Table 28
Indicators of Educational Aspects

Level of Development of ROER
Using the incidence of indicators of reuse and educational issues in ROER, we could identify the level of development of these repositories and establish four main types (Table 29).
A. Top ROER: those that include 9 or 10 of the indicators (4 ROER). Only a few ROER intensively promoted reuse and educational aspects: Merlot (all 10 indicators) and OER Commons, UNIT and TEMOA (all three covered 9 of the 10 indicators).
B. High ROER: those that include 7 or 8 of the total indicators (9 ROER C. Medium ROER: those that include 5 or 6 of the total indicators (31 ROER). The pattern is similar to that of High ROER, but the figures are considerably lower for some indicators, such as educational metadata.
D. Low ROER: those that include 1 to 4 of the total indicators (66 ROER). In general, the figures for all the indicators were much lower, and in particular they lacked (or had more deficiencies in) open formats, granularity, learning goals and educational standards.
Forty-four ROER (40% of the total) were placed in the top, high and medium categories, with different grades of promotion of reuse and educational features. However, the remaining 66 ROER, which make up the bulk of the population (60%), do not have enough drivers to facilitate reuse and educational issues.

Conclusions
Based on the analysis, it has been found that most ROER on higher education are institutional (mainly created by universities and government bodies) and exclusively designed for educational resources, rather than hybrids that also contain research content. Considering that OER production remains essentially linked to public funding, international projects and institutions (Castaño et al., 2016), it makes sense that OER production has no place outside of institutional funding. This is particularly true when there are few rewards for OER In addition, quality and granularity began to play a more prominent role. Therefore, OER-exclusive repositories include a wider range of features, and use them more intensively, to facilitate the reuse of their resources.
The most prevalent educational aspects offered in ROER are content type, format and subject. However, less than half of ROER provide specific metadata related to education, and just over one quarter include learning goals. In addition, the few ROER that provide educational metadata do not always use a specific educational standard such as LOM or SCORM, which were developed to better identify the pedagogical properties of OERs. However, some do incorporate LOM elements in other default metadata specifications, such as Dublin Core. This could be because many institutional repositories use DC as a general schema for describing their resources, since they were created to store not only educational material, but also research or other types of content.
We could assume that educational features are more present in OER-exclusive repositories, which are created to meet an educational need. However, such repositories are not currently achieving their fullest potential. Although there is more educational information in this kind of repository (just over sixty per cent), there are still many cases of OER described and retrieved by type or format instead of by detailed educational metadata that better meets the users' needs, for example the suggested use of content, its duration, pedagogy, target audience or learning goals.
To sum up, current ROERs include more drivers that promote the reuse of OERs, mainly through open licenses and social networks, than features facilitating the retrieval and use of OERs according to educational needs, such as learning goals.
A level of development of ROER was identified, based on the incidence of reuse and educational indicators. We identified four clusters of ROERs ranging from a small top level of repositories that include most of the reuse and educational dimensions to a large lower level of repositories that lack important features such as open formats, granularity and educational metadata. In the middle of this scale, we also identified a high category mainly characterized by providing social networks and educational metadata standards, and a medium category offering a diverse mix of indicators.
We summarize here some of our findings in relation to previous studies. Some similar trends were identified in the general description of the ROER: most of them were multidisciplinary, institutional, predominantly based in Europe and the USA, and supported by DSpace technology. However, there are also a few fine distinctions, in three particular areas: in the numbers, insofar as our data show a significant increase in institutional ROER; in location, given that Europe and especially the UK, Spain and France took precedence over the USA in our study, while other related work found the opposite situation; and finally in technical infrastructure, where in line with other recent studies and in contrast to previous studies, our findings show an emerging tendency to adopt CMS.
Some differences were also observed with respect to prior works. While our results showed a clear predominance of Dublin Core above other specifically educational metadata specifications, other studies found an opposite pattern or simply did not specify the metadata schema used by the ROER.
Other differences observed in our research were increased openness in ROER and a more intensive use of CC licenses overall in areas with higher numbers of ROER, such as Europe and the USA.
Finally, some of our main contributions were that OER-exclusive repositories led to better reuse and educational outcomes than hybrids and that most of the CC licenses used in ROER allowed reuse, as there was a notable presence of BY-NC-SA and BY-NC. This achievement, added to the use of other kinds of open license, shows increased openness of ROERs.

114
To further develop the research, the evaluation framework could be expanded by adding features, and the categorization of ROER could be examined in greater depth. The research could use indicators that are more focused on pedagogy and bottom-up approaches, as these increase the educational usefulness of ROER and the reuse of OER. Additional features could be based on users' interaction, meaning communities and personal spaces, kind of information shared, social tagging procedures, etc., and types of OER, defined by whether the material was produced by students or educators and who it is aimed at (students, teacher trainers, the OER community, learning innovators). Users' participation helps to evaluate and license the stored OER, which facilitates reuse and makes it easier to integrate educational needs into the classification and retrieval process.