Conceptual Framework for Parametrically Measuring the Desirability of Open Educational Resources using D-Index

Open educational resources (OER) are a global phenomenon that is fast gaining credibility in many academic circles as a possible solution for bridging the knowledge divide. With increased funding and advocacy from governmental and nongovernmental organisations paired with generous philanthropy, many OER repositories, which host a vast array of resources, have mushroomed over the years. As the inkling towards an open approach to education grows, many academics are contributing to these OER repositories, making them expand exponentially in volume. However, despite the volume of available OER, the uptake of the use and reuse of OER still remains slow. One of the major limitations inhibiting the wider adoption of OER is the inability of current search mechanisms to effectively locate OER that are most suitable for use and reuse within a given scenario. This is mainly due to the lack of a parametric measure that could be used by search technologies to autonomously identify desirable resources. As a possible solution to this limitation, this concept paper introduces a parametric measure of desirability of OER named the D-index, which can aid search mechanisms in better identifying resources suitable for use and reuse.


Résumé de l'article
Open educational resources (OER) are a global phenomenon that is fast gaining credibility in many academic circles as a possible solution for bridging the knowledge divide. With increased funding and advocacy from governmental and nongovernmental organisations paired with generous philanthropy, many OER repositories, which host a vast array of resources, have mushroomed over the years. As the inkling towards an open approach to education grows, many academics are contributing to these OER repositories, making them expand exponentially in volume. However, despite the volume of available OER, the uptake of the use and reuse of OER still remains slow. One of the major limitations inhibiting the wider adoption of OER is the inability of current search mechanisms to effectively locate OER that are most suitable for use and reuse within a given scenario. This is mainly due to the lack of a parametric measure that could be used by search technologies to autonomously identify desirable resources. As a possible solution to this limitation, this concept paper introduces a parametric measure of desirability of OER named the D-index, which can aid search mechanisms in better identifying resources suitable for use and reuse.

Introduction
Open educational resources (OER) are fast becoming a global phenomenon, which provides hope for bridging the knowledge divide among the masses (Geith & Vignare, 2008). With increased funding and advocacy by governmental and nongovernmental organisations buttressed by generous philanthropy, many OER repositories boasting a large volume of quality resources have mushroomed over the years. With the movement gaining credibility among many an academic community and with the drive toward opening up knowledge for the benefit of the less fortunate taking centre stage (Johnstone, 2005), these repositories have grown rich in knowledge. However, this has in turn given rise to the new challenge of locating resources suitable for use and reuse from the large number of disconnected and disparate repositories available around the globe (Geser, 2007).
As discussed by Hilton, Wiley, Stein, and Johnson (2010) the use and reuse of an OER depends on two factors: the permission and the technologies needed. The authors introduce the four Rs of openness and the ALMS analysis, which can be used to effectively gauge these factors for identifying the most suitable OER for use and reuse. However, at present, all of the three types of OER repositories, which include content OER repositories, portal OER repositories, and content and portal OER repositories (McGreal, 2010), consider only the relevance of a resource to the search query when locating internal and external resources.
Thus, the rank of the search result is not a direct indicator of the suitability of a resource as it does not take into consideration the permission nor the technologies needed to successfully use and reuse. This challenge is further heightened by the common use of OER formats such as PDF, which renders resources useless with respect to reuse (Baraniuk, 2007), and the inability of average users to use the available technological tools to remix the resources (Petrides, Nguyen, Jimes, & Karaglani, 2008). Additionally, as resources are constantly added to these repositories (Dholakia, King, & Baraniuk, 2006), a static method of defining the suitability for use and reuse within the metadata becomes an impossible task.
As a possible solution to this issue, this paper introduces the concept of desirability of a resource, which parametrically takes into consideration (i) the level of openness with respect to the copyright license, (ii) the level of access with respect to technologies, and (iii) the relevance with respect to search rank. The desirability of an OER is then expressed as the D-index which allows search mechanisms as well as users to make informed decisions with respect to the most desirable OER for their needs.

Rationale
In the academic community, the perceived quality of an academic publication or a resource is largely governed by peer review. However, with the present day influx of research publications being made available online, the peer-review mechanism becomes inefficient as not all the experts can review all the publications. As such, an alternative method of measur-ing the quality of a publication or a resource is needed. According to Buela-Casal and Zych (2010), If an article receives a citation it means it has been used by the authors who cite it and as a result, the higher the number of the citations the more utilized the article.
It seems to be an evidence of the recognition and the acceptance of the work by other investigators who use it as a support for their own work.
Therefore, at present the number of citations received is widely accepted as an indication of the perceived quality of an academic publication or resource.
As the styles of citation for academic publications are very well established, search mechanisms such as Google scholar (see http://scholar.google.com) have a usable parametric measure for providing an indication of how useful a publication would be for one's academic research. Although there are established styles of citation and attribution for OER as well, these styles are not standardised or widely practiced when using, reusing, remixing, and redistributing OER. As such, it is extremely difficult for a search mechanism to autonomously identify the number of citations or the number of attributions received by a particular OER material. This issue is further amplified as not all the OER repositories available over the Internet are searched and indexed by popular search mechanisms. Providing potential solutions to this issue are systems such as AnnotatEd (Farzan & Brusilovsky, 2006), which uses web-based annotations, use of brand reputation of a repository as an indication of quality, allowing users to review resources using set scales (Hylén, 2005), and the "popularity" in the Connexions repository, which is measured as percentile rank of page views/day over all time. Despite these very specific methodologies, there is still no generic methodology available at present to enable search mechanisms to autonomously gauge the usefulness of an OER for one's teaching and learning needs.

Definition
The usefulness of an OER for a particular teaching or learning need can only be accurately assessed by reading through the content of the resource. As this is quite a subjective exercise due to one's needs differing from another's, it is extremely difficult for a software-based search mechanism to provide any indication of this to a user. This aspect of use and reuse of OER will remain a human function regardless of the improvements in technology. When considering the use and reuse of an OER, there are other aspects of a resource that are fundamental to the usefulness of that particular resource and can be parametrically identified by a software-based mechanism. The first aspect is whether a resource is relevant to a user's needs. This can be assessed by the search ranking of a resource when searched for with a search mechanism. The search mechanism will compare the title, description, keywords, and sometimes the content of the material to find the best match for the search query. The 62 with the resource. The third aspect is the accessibility of the resource with respect to technology. If the user cannot easily use, reuse, and remix a resource with available technology, the resource becomes less useful. Therefore, the usefulness of an OER with respect to (i) the level of openness, (ii) the level of access, and (iii) the relevance can be defined as the desirability of an OER, indicating how desirable it is for use and reuse for one's needs. Within the requirement of being able to use and reuse a particular OER, these three parameters can be defined as follows: 1. level of openness, the permission to use and reuse the resource; 2. level of access, the technical keys required to unlock the resource; and 3. relevance, the level of match between the resource and the needs of the user.
As each of these mutually exclusive parameters are directly proportionate to the desirability of an OER, the desirability can be expressed as a three-dimensional measure as shown in

The Scales
In order to parametrically calculate the desirability of an OER, each of the parameters discussed above needs to be given a numeric value based on a set scale. These scales can be defined in the following ways.
The level of openness can be defined using the four Rs of openness (Hilton, Wiley, Stein, & Johnson, 2010) as shown in Table 1. The four Rs stand for reuse, the ability to use all or part of a work for one's own purposes; redistribute, the ability to share one's work with others; revise, the ability to adapt, modify, translate, or change the form of a work; and remix, the ability to combine resources to make new resources. The values 1 to 4 were assigned to the four Rs where 1 corresponds to the lowest level of openness and 4 corresponds to the highest level. Remix 4 The level of access was defined on a scale of 1 to 16 using the ALMS analysis (Hilton, Wiley, Stein, & Johnson, 2010), which identifies the technical requirements for localisation of an OER with respect to access to editing tools, level of expertise required to revise or remix, ability to meaningfully edit, and source-file access. As shown in Table 2, the value 1 corresponds to the lowest accessibility and value 16 to the highest accessibility. The relevance of a resource to a particular search query can be measured using the rank of the search results. According to Vaughan (2004) users will only consider the top ten ranked results for a particular search as the most relevant. Vaughan further suggests that users will ignore the results below the top 30. Based on this premise, the scale for the relevance was defined as shown in Table 3, where the value 1 is the least relevant and value 4 is the most relevant. Calculation Based on the scales, the desirability of an OER can then be defined as the volume of the cuboid, as shown in Figure 2, calculated using the following formula. desirability = level of access x level of openness x relevance As a result, the desirability becomes directly proportionate to the volume of the cuboid. By normalising the values indicated in Table 1, Table 2, and Table 3 to make the scales uniform for the calculation, the D-index of an OER can be calculated using the following formula. D-index = (level of access x level of openness x relevance)

/ 256
Based on the above calculation, a resource becomes more desirable as the D-index increases on a scale of 0 to 1, where 0 is the least desirable and 1 is the most desirable.

Verification of Concept
The most commonly used method for locating OER is to use a generic search mechanism such as Google or to use a search mechanism specific to an OER repository such as Connex- ions (see http://cnx.org/) or Wikieducator (see http://wikieducator.org). However, both of these types of search mechanisms only consider the relevance of the resource either by matching the title and description or the keywords to the search query provided by the user.
Therefore, the resources returned as the top search results might not always be the most desirable for use and reuse in a given scenario as they might be less open or less accessible.
The D-index is specifically designed to overcome this limitation by taking into consideration the openness and the accessibility of an OER in addition to the relevance to the search query. When applying the D-index to an OER repository, the level of access, discussed in

Calculation of the D-index
To demonstrate how the D-index was calculated for each search result, a general search was conducted on the OER Commons repository for the term calculus using its native search mechanism. Out of the 165 resources returned as results, three resources at the postsecondary level with different search rank were chosen for comparison as shown in Table 5.

MS Word
The search rank, licence, and the file type of each resource in Table 5 was then compared   with Table 3, Table 4, and Table 2 respectively to identify the parameters required to calculate the D-index as shown in Table 6. Looking at Table 6 we can see that the search mechanism has ordered the results according to the relevance where resource A is the most relevant. However, resource A is less open and less accessible when compared with resource B. Table 7 shows how the results would be reorganised when the D-index is applied to the same search results.  Table 7, it can be seen that resource B would be the most desirable OER for use and reuse due to its level of openness and access even though resource A was the most relevant. Table 8, Table 10, and Table 12 show the top 10 results returned by the native search mechanisms of MERLOT, JORUM, and OER Commons respectively for the keyword calculus.      Table 8 and Table 9, which show the search results returned by MERLOT, it can be seen that the original top 10 search results (Table 8) only contain resources that are released under the CC BY-NC-SA license. This license significantly restricts the user's freedom with respect to the four Rs. Also six of 10 resources returned are in PDF format, which make them difficult to reuse and remix. It must also be noted that the resource ranked as number 10 is a protected resource, which requires a specific username and password to access. Looking at Table 9 where the results are reranked according to the D-index, it can be seen that eight of 10 resources are in HTML/text formats, which are the most accessible in terms of reuse. Four of 10 resources are available under the CC BY licence, which make

Experiment Results
them the most open resources in the list. Similarly, by comparing Table 10 and Table 11, we can see that the use of the D-index has reranked the top 10 results so that the most accessible resources are ranked at the top instead of resources that use proprietary software applications. The video resources returned were given an accessibility value of 12 according to the ALMS, where access to editing tools = high; level of expertise required to revise or remix = high; meaningfully editable = yes; and source-file access = yes.
Analysing Table 12 it can be seen that four of 10 results returned by the OER Commons search mechanism are copyright protected. As such these cannot be considered as OER and are the least useful for a user who is searching for open material. A value of 0 for openness was assigned to these resources during the D-index calculation. Furthermore, five of the top 10 results returned by the OER Commons search mechanism were in PDF format. Looking at Table 13

Application and Limitations
The D-index can be incorporated into any search mechanism of an OER repository provided that the resources in the repository are appropriately tagged with the necessary metadata, such as title, description, keywords, copyright license, and file type. Many OER repositories now require authors to define the basic metadata, such as the title, description, keywords, and copyright license. As such, the use of these parameters to gauge the values for relevance and openness becomes an easier task. However, gauging the access parameter which uses the file type of the OER becomes a much more challenging task as some resources consist of multiple files of multiple formats. This can be rectified by breaking a collection of OER into individual learning objects, which allows software applications to determine the file type of the individual OER.
A couple of practical limitations can also be identified with respect to the implementation of the D-index in OER repositories. One of these limitations is that the desirability becomes one dimensional due to the copyright license and the file format being fixed in repositories such as Connexions or Wikieducator. As a result, the D-index becomes only a function of the relevance parameter which does not add much value to the existing search mechanism.
Therefore, the D-index is best suited for use in portal repositories/content and portal repositories, such as the OER Commons, MERLOT, and JORUM, which have a wide variety of resources of different file types released under various copyright licenses. It will also be quite effective when used with search mechanisms which query multiple repositories to identify resources. to effectively locate useful resources for specific teaching and learning needs from the variety of disconnected and disparate repositories available. This gives rise to the challenge of identifying a parametric measure of the usefulness of an OER, which will enable users to effectively identify suitable resources without reading through countless unsuitable ones.
The concept of desirability of an OER introduced in this paper attempts to lessen the pain of OER users with respect to identifying resources that are relevant, open, and accessible for one's particular needs. Currently, users who search for OER in specific repositories use search mechanisms native to the repository to identify relevant resources. Depending on the algorithms used by the native search mechanisms, the search query will be compared against the metadata of a resource such as title, description, and keywords to provide a list of resources which might be of relevance. However, these search mechanisms do not take into consideration the level of openness or the technological skills required with respect to using, reusing, remixing, and redistributing a resource. The D-index is an attempt to factor in the openness and accessibility in addition to the relevance in order to provide OER users a useful set of search results which are appropriate to their needs.

74
The D-index can be incorporated into any OER repository provided that the necessary metadata for calculation are available. It is most effective when used in portal repositories/ content and portal repositories which search multiple disconnected OER repositories to locate relevant material. The greatest benefit of the D-index to teachers and learners is its ability to locate and list the most desirable OER for use and reuse from the numerous combinations of relevance, openness, and access under which OER are released. The authors are currently working on incorporating the D-index into an artificial intelligence (AI) based text mining system named OERScout which is used to cluster OER available in all the disconnected repositories based on autonomously identified keywords. The use of the D-index in this clustering process will enable search mechanisms to effectively locate OER which are most desirable for use and reuse.