Evaluating NTU’s OpenCourseWare Project with Google Analytics: User Characteristics, Course Preferences, and Usage Patterns

As freely adoptable digital resources, OpenCourseWare (OCW) have become a prominent form of Open Educational Resources (OER). More than 275 institutions in the worldwide OCW consortium have committed to creating free access open course materials. Despite the resources and efforts to create OCW worldwide, little understanding of its use exists. This paper reports OCW project development at National Taiwan University (NTU) and investigates its use with Google Analytics. Reports include strategic plans to overcome challenges to OCW creation and implementation, the project’s growth and maturation, overall use of OCW, and possible future directions. As a result of its 5-year development and of lessons learned, the NTU-OCW experience features: (1) integrating resources on campus and established operating procedures, (2) setting course selection criteria featuring the strength of NTU and Taiwan, (3) providing coherent program support to enhance faculty participation, and (4) adhering strictly to the Creative Commons license. Data from Google Analytics was reviewed for better understanding of the use, characteristics, course preferences, and behaviors of NTU-OCW users. Results show visitors were primarily lifelong learners (65%) in informal learning settings. Statistics indicate an overall successful use of NTU-OCW for Chinese speaking users, especially in urban areas where information and communication technology is more developed. Potential impacts and future improvements are discussed, including how to promote usage of OCW courses for on and off campus users, adding rating features and indexing for customizing search, and integrating OCW into the learning management system (LMS) as part of OER.


Introduction
In the global movement of Open Education Resources (OER), OpenCourseWare (OCW) is an important force. An OCW is defined as "a free and open digital publication of high quality educational materials, organized as a course" (Carson, 2009, p. 27). These resources are "free and openly licensed, accessible to anyone, anytime via the internet" (Open Education Consortium, n.d.-a, para. 1). Since the Massachusetts Institute of Technology (MIT) initiated its OCW project in 2001 and inspired others to follow, 275 institutions around the world have joined the Open Education Consortium (OEC) to share the common values of openness, equity, collaboration, and multiculturalism (Open Education Consortium, n.d.-b).
However, like every educational movement, every OCW project differs from one to another due to unique educational settings. To better promote access to education, to improve the quality of education, and to empower both well-served and underserved populations through the power of open learning, regional consortiums or organizations formed over the years. OEC sustaining members include among many others the African Virtual University (AVU) for Pan African, Universia (a Latin American, Spanish and Portuguese network), Japan OCW consortium, Korean OCW consortium, and Taiwan Open Course Consortium in Asia.
As one of Taiwan's leading universities, National Taiwan University (NTU) is known for its high-quality teaching, learning resources, and richness of course content. NTU also shares societal responsibilities in providing quality education, conducting higher level research, and promoting open education models for others to follow. Inspired by and joining the world-wide open education mission, NTU adopted the OCW concept to promote open education and to share knowledge by making educational resources available through the Internet. In November 2008, NTU joined the Taiwan Open Course Consortium (TOCC), and began research and discussions on developing a platform for open educational resources. As a result, NTU began the OCW project in 2010. In July 2011, NTU became a member of the OCW Consortium (now Open Education Consortium) and was committed to developing an OCW website with high quality course materials. Through OCW, NTU can share its educational resources with the public, and enhance the participation of teachers and students in creative teaching and learning. With its quality-first principle, consistent effort, and collaboration from all involved individuals, the NTU OCW site reached the 12-millionvisitor mark in 2016. In addition, its website was named the "Outstanding Site" by OEC in 2014. During the following year, an NTU OCW faculty member received the "Educator Award" of the Open Education Awards for Excellence.
However, realizing the great potential of OCW concept requires consideration of many issues. Some are global, and some are unique to individual institutions or to specific geographic regions. While most previous studies about OCW focus on initiatives in a western context, there is a need to explore and share successful OCW experiences in Asia, specifically in Taiwan. From the perspective of NTU-OCW, several questions should be answered: Would the OCW concept fit NTU's core values and characters? What could NTU best offer, considering its unique position in Taiwan and the world? To what extent has it used? By whom and how? How could the greatest benefit with limited resources be provided locally and globally? As a case study, this paper presents the situations NTU faced while adopting the OCW concept in creating open educational resources. The paper shares NTU's experiences with implementation and promotion of online learning materials, specifically OCW courses and discusses OCW use and provides suggestions for producing high quality online learning resources.
102 This paper will introduce key concepts from the literature related to open education, OER, and OCW. Next, it will detail the development and implementation of the NTU OCW experience. Google Analytics will provide insights into the uses, characteristics, course preferences, and behaviors of NTU-OCW users.
Finally, the implementation of findings and future direction will be discussed.

Literature Review
Discussions about OER have accelerated during the past decade and related initiatives are being launched around the world (Guo, Zhang, Bonk, & Li, 2015). Previous research is discussed from four perspectives: (1) OER, (2) OCW, (3) OER, and OCW in Taiwan.

Open Education Resources
Rapidly developing Internet technology continues to bring new opportunities to education. One promising and exciting example is OER. While the term "open educational resources" was first adopted by UNESCO in 2002, the idea of open education emerged in the late 20 th century from developments in distance learning.
The culture of open knowledge and open sharing allows easy access and easy reuse of teaching and learning materials to a wide range of educators and learners. OER typically refers to digital resources, including articles, e-text, video, audio, modules, full or partial courses, and any learning materials (Wiley, Bliss, & McEwen, 2014). However, based on UNESCO's definition, OER are "resources for teaching, learning and investigation which are in a public domain site or have been published under an intellectual property license, allowing other people its free use or for different purposes than those referred by its author" (Peláez, Pullaguari, & Caro, 2011, p. 1075 (Educause, 2010). OpenCourseWare, Connexions, and the Open Learning Initiatives are some of the well-known OER projects.
Given the philosophy of OER to be open and make education more accessible, OER is one way to address the rising costs of education (Educause, 2010;Wiley, Green, & Soares, 2012). Many recent studies have discussed the potential of OER materials and resources replacing traditional paper-based textbooks.
Hunsicker-Walburn, Guyot, Meier, and Beavers (2016), for instance, investigated community students' preferences of OER textbooks. More than half the students (64%) indicated textbook costs would impact their perception of attending higher learning. In addition, in their study, approximately 93% students felt satisfied with OER textbooks. Similar results were found in Delimont, Turtle, Bennett, Adhikari, and Lindshield (2016) which surveyed and interviewed students and faculty in 13 university courses that used OER. They found that all students and nearly all faculty had positive preferences toward using OER. Hilton (2016) reviews 16 additional studies pertaining to OER efficacy and perceptions and find that generally, OER can be successfully substituted for traditional learning resources. In another study, Watson, Clouser, and Domizi (2014) shared redesign strategies, financial benefits, and pedagogical opportunities from two 103 courses that have replaced textbooks with OER. They concluded that collective advantages provide a compelling argument for broad adoption of OER in higher education.
Despite such positive research results related to the use of OER in instruction, several issues emerged with the increasing volume of OER creation and its usage. Such issues include the need for effective search methods to find a proper or suitable OER, quality control of OER creation, and evaluation mechanisms for OER. Groom (2013) indicated that decreasing the technical and metadata requirements of finding and sharing OER can enhance teachers' levels of participation in OER. Given that anyone can create OER, the quality of the OER is not always consistent. In most cases, OER are developed by individuals who may not have proper training in instructional design or who participate in fairly casual and unstructured online collaboration (Daniel, Kanwar, & Uvalic-Trumbic, 2009). In addition, Dhanarajan and Abeywardena (2013) find that faculty member's knowledge and skills of evaluating OER quality can impact their adoption and use.

OpenCourseWare
Although OER and OCW are sometimes used interchangeably in studies, the two differ. OCW usually includes syllabus, handouts, course slides, lecture videos, and other teaching and learning materials that are "organized as courses." OER resources can range from "any digitized form" (Terrell & Caudill, 2012, p.39.). Therefore, OCW is part of a larger subset of OER. In the global OER movement, the MIT OCW has continued to play an essential role since its inception in April 2001 (MIT, 2001).
Today, MIT OCW stands as one of the longest-running OER projects in the world (Educause, 2010). While several leading universities considered offering distance learning programs to a world community for profit by relying on their famous brands and resources, the MIT OCW project created a non-profit model for open knowledge and open sharing (Carson, 2009;Shumar, 1997). In June 2001, MIT announced the launch of two projects to make approximately 2,000 lecture materials available online with an open license. That means learners around the world can access high-quality course resources that are "free and openly licensed, accessible to anyone, anytime via the internet" (Open Education Consortium, n.d.-a). To date, MIT OCW website has 200 million visits from users around the world. Among these users, 43% identified themselves as self-learners, 42% as educators, and 9% as students (MIT OpenCourseWare, 2016, para. 1). The dual missions of MIT OCW clearly expressed the concept of OCW, "to provide free access to virtually all MIT course materials for educators, students and individual learners around the world and to extend the reach and impact of MIT OpenCourseWare and the OpenCourseWare concept" (Carson, 2009, p. 26). OCW shares educational resources and course materials to anyone who wants and needs it, and the OCW movement promotes the idea of knowledge sharing itself.
The growth of OCW inspired other institutions around the globe to join the OER movement. The Open Education Consortium (former OCW consortium) was established in 2005 and was intended to reintegrate educational resources, promote higher education and contribute to global knowledge sharing. Following that model, regional consortiums began to emerge including Universia to offer the Latin American, Spanish, and Portuguese community, African Virtual University (AVU), the leading Pan-African Open Distance and eLearning Network to facilitate the use of open, distance, and eLearning in African institutions, as well as Japan OCW Consortium and Taiwan Open Course Consortium (TOCC) in Asia.
While the idea of OCW was broadly accepted around the world, its actual usage and impact on learning creates some doubts. A common criticism of OCW is its one-way delivery of information. Although content materials are organized into courses, OCW lacks interactions among teachers, students and peers common in classrooms. Cueva, Rodriguez, and Marban's (2016) study used quantitative and qualitative methods to investigate the level of acceptance and expectations of integrating social technologies in OCW. They concluded that with implementation of collaborative learning design and supporting technologies, the effectiveness of social and collaborative learning in OCW can be enhanced. Massive Open Online Course (MOOC) that provide certain kinds of interactions between students with teachers and among peers, or Small Private Online Course (SPOC) that combine OCW and classroom are additional ways to use OCW materials interactively.
Many important issues for OER and OCW include design, development, accessibility, suitability, distribution, selection and quality control. One major issue of concern for both OER and OCW is the quality of materials. Given the increasing creation, availability, delivery, and use of OER materials and OCW courses under the concept of educational openness, two emerging issues are how to build a quality model and how to designate criteria for online learning content. Many previous studies focus on the effectiveness and quality aspects of MOOCs. Walker and Loch's (2014) survey with academics who participate in MOOC found that the delivery quality of MOOC does impact academics' perception to MOOC. Video quality, interaction, and assessment are some aspects mentioned in their survey results. Yousef, Chatti, Schroeder, and Wosnitza (2014) examine the criteria of effective MOOC and found that among all the criteria, learning analytics and assessment were identified as the most important ones. Among all the quality assurance frameworks for online learning that are proposed by various researchers, one of the most popular and widely used is the Quality Matters Program (QM). QM suggests eight quality dimensions for online learning: course overview and introduction, learning objectives, assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, and accessibility. However, because these factors are for general online learning, they might not be suitable for OCW due to its unique features. For example, OCWs are delivered in a synchronized environment and usually provide only one-way interaction of information. Evaluation or assessment elements are not included.
An OCW study indicated that economic consideration and the ordering of intellectual property rights are two challenges to its adoption (Peláez et al., 2011). Tovar, Piedra, Chicaiza, Lopez, and Martinez-Bonastre In their paper about proposing a quality model for OCW materials, Peláez et al. (2011) identified three essential areas that OCW materials must address: educational, technical, and legal. Educational aspects refer to "teaching materials structured in adequate way with a common appearance. " Technical aspects include "an Internet site with worldwide access and with the proper quality." Legal aspects consider "the open licenses application and the use of materials in a non-commercial context." (p.1076) For effective OCW adoption, economic consideration and the ordering of intellectual property rights are the two main challenges.
The increasing numbers of OCWs do not assure their high quality or educational utility (Porcello & Hsi, 2013;Rodríguez, Dodero, & Alonso, 2011). Moreover, low-quality OER/OCW can hamper instructional practices and waste teachers' limited time (Abramovich, Schunn, & Correnti, 2013;Wetzler et al., 2013). As a result, developing high quality OER has become a critical factor in the success of online education. TOCC, established in 2008, aims to expand open education to all fields of studies and to break down university walls and boundaries to knowledge exchange. As a sustaining member of OEC, TOCC promotes OCW under the principles of openness, sharing, and collaboration, and provides diverse and rich learning resources for global Chinese communities. By the end of 2016, TOCC consisted of 28 members including 27 higher-education institutes and one senior high school. One of the features of TOCC courses is that more than 75% are full-courses, which include lecture videos and required course materials (Wan & Lee, 2012).

OER/OCW in Taiwan
In response to the OCW wave or OCW inspiration to open education, related issues were discussed from various perspectives in Taiwan. Wu (2014) examined 11 empirical studies about OCW in Taiwan that were published in Chinese. Primary topics included advantages and disadvantages of adopting OCW, OCW applications in various subject areas, such as calculus (Chen & Pai, 2012), computer skills (Tsai, Shen, & Huang, 2012), and teacher education (Wu, 2014). Topics on successful courses, designed and development of OCW websites, intellectual property concerns, and the perceptions of faculty and students were discussed (Wu, 2014). Chou, Hung, Wei, and Wang (2011) probed faculty members' readiness for OCW in Taiwan using an 8factor scale, including acceptance of OCW, personal characteristics, self-perceptions of ICT competency, teaching method and style, effect on teaching and research, administrative support and incentive, intellectual property issues, and interaction with users. In addition, Wang, Chen, and Hu (2013) interviewed faculty members in nine Taiwanese public universities and found that faculty's beliefs of OCW and OER play an essential role in their participation in OCW. They also found that administrative support contributes to a faculty member's willingness to use OCW materials or participate in an OCW project.
While most of the leading universities in Taiwan devoted effort and time in OCW production, related studies are few, especially articles published in English. Among those few studies to date (Chen & Pai, 2012;Chou et al., 2011;Sun & Wu, 2016;Wan & Lee, 2012;Wang et al., 2013;Wu, 2014;Young & Hung, 2014), the researchers have focused on the initiatives of OCW projects and the adoptability/acceptance of OCW in higher education. In addition, there was a shift from OCW to MOOC in literature regarding open education and online learning. In 2013, Taiwan's Ministry of Education launched a nationwide MOOC project in 15 universities (Chen, 2013). NTU also hosted the first-ever Chinese MOOC on Coursera. Since then, the primary focus of OER in Taiwan has shifted from OCW to MOOC. As a result, many universities concentrate 106 on making MOOC and discontinued production of OCW. The focus of research interests also shifted to MOOC related topics.
In recent years, the few remaining OCW studies are usually caused by emerging technologies, such as the integration of mobile devices in OCW. Young and Hung's (2014) study discussed implementation of a mobile-based OCW platform and its usages. They concluded that to expand OER in Chinese societies, issues about intellectual property rights need further attention. They also argued that cloud-based platforms are one of the more important and interesting approaches for future OCW initiatives and projects. Another area of recent interest and attention is the effect of flipped classroom approaches to OCW (Sun & Wu, 2016;Sun, Wu, & Lee, 2017). Sun and Wu's (2016) study with 181 freshmen in a course on physics found that the integration of OCW and flipped classrooms can help students achieve higher learning performance than the integration of OCW and a conventional classroom. In spite of the importance of these studies, few have focused on the use of OCW project and the evaluation of design and solutions in responding to local or global concerns or needs.
Currently, NTU is one the few universities that continuously produce both OCW and MOOCs. Although there are studies discussing OCW issues in Taiwan, most were conducted before 2013 and the majority of them are published in Chinese. To understand the current state of OCW in the post-MOOC era in Taiwan, it is necessary to take a look at NTU OCW for updated information. It can provide a good reference to the development of OCW in non-western context.

Purpose of the Study
In spite of the importance of the previous research, none has focused on the use of OCW and an evaluation that could inform design and solutions in response to concerns and needs globally or locally. In addition, although the popularity of NTU OCW is continuously increasing and the quality of materials are recognized by peer institutions and organizations in the OER community, its current usage and factors leading to the achievement are rarely discussed. Therefore, the purpose of this paper is twofold. One is to contribute to the gap of literature about the use of OCW and the other is to examine the possible reasons contributing to the success of NTU case. Together, this paper offers findings that inform researchers and educators about the potential next steps of OCW projects and initiatives at NTU and hopefully to address a more global audience.

Significance of the Study
While many efforts have made to create OCW, there is little understanding about how OCW was used. The present paper provides information about the use of OCW. It fills the gaps in literature about how OCW was used. This paper also provides various suggestions for future directions, based on the findings from qualitative insights and quantitative google analytic data. We believe "next steps' are critical for the OCW community after achieving its initial goals, to develop and implement OCW since its MIT inspiration. As such, the present paper can be a reference or a model for others on "what next" or facing similar challenges, including suitability, roles in open education, and design for better impact on teaching and learning. It is especially important for the OER community in Taiwan.

Development and Implementation of NTU OCW
This section details the development and implementation of the NTU OCW experience, including: (1) the context of NTU-OCW, (2) the formation of NTU-OCW working group in responding to the call for action and challenges, (3) the features of NTU-OCW experience that resulted from the process, and finally (4) the NTU-OCW production model that leads to OCW success.

Context of NTU-OCW
National Taiwan University, one of the most prestigious universities in Taiwan It became apparent that a task force and a model were needed to overcome these issues and to carry out the plans. One immediate action was to form an OCW working group in 2011 to coordinate tasks between and among multiple units across the university. The same group is responsible for carrying out OCW plans, resulting in the current NTU OCW site/project.

NTU-OCW Working Group
Conversations between and among multiple units on campus revealed a need to have a unified unit or working group to efficiently and effectively utilize university resources as well as enhance/ensure course quality (although it is access-free). The primary role of this working group is to ensure success of the NTU OCW project. This involves multiple layers of planning, organization, and execution. Strategic planning and execution tasks include:  Course selection criteria and quality standards. 108 CASE, College of Liberal Arts, NTU press, and CTLD are responsible for course creation and production.

DCIM, the Division of Computer Information Management in the Office of Academic Affairs, is in charge
of development and maintenance of the project website (see Figure 1). The Division of Learning Technology (DLT), a sub-unit in CTLD, was primary working unit dedicate to OCW project. Besides in charge of approximately 80% of OCW course production, the unit coordinate tasks between course production and the website, provide instructional design consultation to faculty and student production assistants (PA) training. (See Figure 1) The project group meets monthly to track working progress and to discuss issues related to effectively integrating cross-unit manpower to allow the project to move forward quickly. In addition, standardized production processes for inviting and producing courses were established, in the early stages of the project, so group members could follow implementation. Furthermore, these standards were used as the basis for tracking and evaluation of the project to ensure the effectiveness of its implementation.

Features of NTU OCW Implementation
The features of NTU OCW are the result of strategies used for overcoming challenges or barriers while implementing OCW at NTU.

1.
Integrating existing resources and establishing standard operating procedures.

109
Like most OCW projects, adequate financial and other resources are major challenges. To start the project, NTU OCW integrated all existing resources in the university, including administrative support, multimedia production resources (skills, manpower and equipment), and professional expertise on internet intellectual property. As a result, the NTU OCW working group, a unified cross-unit project group, was formed to coordinate efforts and carry out the mission. For the same reason, operating procedures were created to ensure quality and smooth workflow between units.

Setting course selection criteria featuring the strength of NTU and Taiwan.
With limited sources and an unwillingness to compromise quality, NTU-OCW needed to be selective in terms of the course selection. Rather than open to all courses, NTU-OCW developed course selection criteria that feature the strength and specialty of NTU, such as "Dream of Red Chamber" in classic Chinese literature, "I-Ching," "Zhuangzi" in Chinese philosophy, and Sociology of Love. These courses were the most popular on campus. The quality of courses and scholarship of the faculty instructors have been recognized in appropriate subject areas. The criteria also consider what would benefit the largest audience, such as general education courses, core courses, and fundamentals in several professions. In addition, NTU OCW invited distinguished teaching faculty members who received NTU outstanding or excellent teaching awards to participate.
3. Providing coherent program support to enhance faculty participation.
Faculty members play essential roles in any course, including OCW courses. Unfortunately, low faculty participation is a common challenge for the creation and adoption of OER. Possible reasons include a lack of related experience or skills (i.e., digital media development), the quality of online courses, intellectual property concerns about course content, and extra workload to an already busy schedule. These are common barriers in most e-learning instruction (online materials, distance programs, and blended learning). Solutions include simplifying production process with templates or models that ensure quality and effectiveness of products, providing support in copyright licensing, offering incentive and grant opportunities for contribution, and moving as much production as possible to the classroom where instructors teach, rather than going to the studio. These steps are intended to reduce teachers' common concerns and increase participation.

Strictly following the Creative Commons license.
One of the important features of OER and OCW is to freely open its content to all learners. To make the materials freely open, accessible, and used by others for educational purposes, Creative Commons license must be implemented. This allows educators and researchers to use the content for educational purposes, to modify it to create more diverse elements and usages, and to disseminate or share it. Under this premise, the intellectual property of OCW course materials should be protected by strictly following relevant laws and regulations. The most common type of CC license requested by faculty members is CC BY-NC-SA, Attribution-NonCommercial-ShareAlike, which lets others remix, tweak, and build upon their work only for non-commercial purpose. As a result, course materials can be adopted in more diverse ways and disseminated/shared to wider usage for educational purposes. All materials were reviewed by the 110 production team, including a copyright lawyer, with the "copyright procedure" before uploading to the OCW site (see Figure 2.) Figure 2. Copyright procedure.

NTU OCW Production Model
The Division of Learning Technology, a sub-unit in CTLD, and CASE are responsible for course production and for dealing with intellectual property materials. To meet goals and overcome challenges, the Center of Teaching and Learning Development established a working model for OCW creations and formed a team designated to carry out the OCW mission. While other operational layers are important, this paper focuses on the production layer, the essential part of the NTU-OCW working model and another key factor in NTU's success. Two important and successful strategies are: (1) established operating procedures, and (2) a designated ID-team approach.
Operating procedure: All NTU OCW courses contain comprehensive content materials including video clips, hangouts, and course materials. High-standard intellectual property regulation and Creative Commons (CC) are applied to all NTU OCW course materials. The creation of one OCW course is a continuing process throughout a semester (or a process of more than 22 weeks, including production assistant (PA) training, 111 faculty consultation, preparation, and wrapping up). All NTU-OCW courses include course contents of a full semester (18 weeks). There is a need for well-designed operating procedures to achieve efficient workflows and organization with multiple tasks/courses across units. Such procedures also ensure the quality of final production materials. Instructional design principles were applied. Instructional consultation to faculty were provided before the semester started to ensure best results in transferring the course to an online environment.
Designated team approach: a designated trained production team was assigned to each faculty for a specific OCW course. A team usually consisted of two student PAs supervised by the CTLD staff with a background in instructional design and in assisting faculty in two major areas: media production and intellectual property/CC. For a 3-credit hour course, the average workload for a student PA is 5-10 hours per week.
Each CTLD staff member typically supervises two or three courses per semester. Participating professors received support for his/her course. Combined with operating procedures, the designated and trained team approach represents the quality control gatekeeper.

Faculty Support
As mentioned earlier, faculty members have essential roles in the completion process. Support and resources were provided to ensure that the participating faculty was comfortable and free of concern.
Common support includes instructional design consultation and intellectual property consultation.
Instructional consultation was provided to explain the production process and to discuss the best way to transfer face-to-face courses into an OCW course delivered online. Faculty was encouraged to use materials best suited to their teaching, including copyright restricted materials, open licensed materials, and resources of their own creation. The production team provided proper support regarding copyright issues.

Student Involvement
To make the OCW project more financially sustainable, NTU OCW recruited part-time production assistants (PAs) to assist in the course production process. Production assistants (PA) are primarily students with job responsibilities in two categories: (1) Intellectual property assistance for assisting professors in creating course content and materials (such as presentation files, and handouts), and to double check content source and copyright notices; and (2) Media production assistance for assisting in the OCW media production process, including set-up, video shooting, and editing of course sessions. This is a key factor in the success of NTU-OCW and in making NTU OCW more financially sustainable. The working experience also provides students with opportunities to gain more understanding of course content.

Students learn technical skills and deepen their knowledge as a result.
Tips for part-time PA recruitment and training. NTU OCW usually suggests professors recommend their own students or students who took the course previously to assist in handling intellectual property.
These students are familiar with course content which can enhance communication with faculty instructors and the efficiency in processing course content and confirming sources. The same concept applies to media PA recruitment. NTU OCW recruits students interested in OER/OCW and have multi-media experience to be media assistants. These students are more likely to complete their tasks successfully.

112
Before starting work, all part-time PAs must complete two stages of training. First, before the beginning of each semester, an all-day training session is required which includes an introduction to open education, the basic concepts of copyright and CC license, an explanation of the work involved, and practical drills. This training usually takes place one week before the semester starts. It allows PAs to complete the training and quickly apply what they have learned in their new tasks. The second stage of training occurs in the first few weeks of the course, when full-time staff/assistants sit in student classrooms to guide PAs toward being able to work independently.

The Use of NTU OCW
This section reports the results of the investigation on the use of NTU OCW. Data from Google Analytics was reviewed to gain a better understanding of the uses, characteristics, course preferences, and behaviors of NTU-OCW users. Specific areas for investigation include the growth of the site, type of users, users' devices, length of viewing, and demographical distribution. Data was collected and analyzed to answer the following questions:  How many courses were created?
 How many visitors accessed the sites?
 What percentages of users/students were new versus returning students?
 What kinds of devices were to access NTU-OCW?
 What were the characters and viewing behaviors of users?
 How can the findings guide NTU for next steps related to OCW and other forms of open education?

Growing Number of Courses, Visitors, Faculty Involvement
Since 2011, 213 high quality OCW course materials were completed and available for users on the NTU OCW website. On average, approximately 43 new courses were added to NTU OCW each year. In 2016, OCW site reached the 12 million visitors mark. More than 100 faculty members have participated in the project. Several devoted professors have more than two courses on the NTU OCW.
Before the NTU OCW working group was established in 2011, NTU had 31 online courses created by different centers and departments, including CASE, GET, and NTU Open Forum for New Intellectuals. New productions were started by the OCW working group in late 2011. In 2013, special attention was given to creating higher quality materials in video, audio, and comprehensiveness of course content. (Figure 3) 113  (Table 1).

Characteristics of OCW Audience (Site Visitors)
In 2016, the NTU OCW website exceeded 12 million visits. On average, there are 9,700 sessions every day (see Figure 4). Visits have increased from 800 to 8,000 daily, a clear demonstration that NTU OCW website's visibility and popularity are growing.

115
More than half (67%) of NTU OCW users are returning visitors, indicating a stable part of the audience continues to view online courses ( Figure 5). It is encouraging to learn visitors are returning, which could indicate continued learning, one of primary missions of the OCW. This is especially true for the "non-school" age group (typically older than 25 years old unless they are pursing advanced degrees). According to GA, the average pages/session for each user is 3.87. This means most users of NTU OCW go almost directly to the course page they want or they leave the site after viewing four pages. Users in the 35-44 age group usually view the most pages (3.50) in each session, and users in the 18-24 age group view the least pages (2.95) in each session (Table 3). There is no difference in average page viewing between male and female users. In addition, the average time/session for each user is 4:35 minutes. As Table 3, shows users in the 18-24 age group spent the shortest time in each session, while elderly users in the 65+ age group spent the longest time in each session. Although there is a difference between age groups, no significant difference is suggested. However, as Figure 7 shows, the percentage of desktop use has decreased significantly and mobile device use has increased significantly in the past five years. When NTU OCW began in 2011, 94% of users used desktops to access the website. Only 3% of users used mobile devices. This trend toward the use mobile 117 devices to access online learning materials suggests that designing for multiple devices must be considered in the future. Figure 7. Devices NTU OCW users use to access the website.

Conclusion
Overall, the NTU OCW was successful despite the many challenges detailed in this study. The emphasis on quality was the primary factor in its successfully attracting visitors and supporting learning. NTU OCW has won the best OCW site by the Global Open Education Alliance in 2014. An OCW professor won the Open

Team Efforts
Team effort is another key factor in success. Regardless of the challenges, issues, and limitations, NTU OCW has achieved its initial goal to successfully provide whole courses with high quality open educational materials. It requires collaboration and team efforts across units, including administrative support from the university, the Dean of the Office of Academic Affairs, the Director of NTU Open Forum for New Intellectuals and the Director of CASE, and CLT. Participating professors' openness and enthusiasm for education to all is also an important factor in project success. More than 100 professors now participate in NTU OCW and share a devotion to open courseware. NTU OCW will continue to meet new challenges going forward.

Future Directions
With support from the university and many participating teachers, NTU OCW achieved most of its initial goals. However, with the recent emergence of MOOCs and various but significant funding reductions, NTU OCW has an urgent need to start thinking of new goals and strategies as its next steps.
Because so much effort, time, and resources have been expended to create a significant number of quality OCW courses, their wide spread use is important. A natural next step will be to promote OCW as valuable OER that can be accessed online freely for educational purposes. For OCW promotion, an array of information will be provided steadily at activities inside and outside of campus. To actively promote OCW courses to students in school, 1-minute promotional clips for selected courses are to be produced and broadcast on digital signage in school buildings and on the NTU channel of YouTube EDU.
These promotional clips usually include short footage devoted to course lectures or images of course materials. Some interesting questions related to the course content are included in the clips, encouraging students to visit the website, find the complete course videos, and obtain answers. Furthermore, in order to reach a wider audience outside the school, pieces of course sessions are also broadcast on the NTU channel of YouTube EDU. Members of the audience who watch the course sessions and want to learn more can be linked to the complete course page on the NTU OCW website. Social media such as Facebook is also used to promote course content and the OCW website. Although many positive responses have been received, the possibility of attracting a new and different demographic audience remains questionable.
For future e-learning, NTU OCW wants to effectively increase the usage of course content. Possible practices may include research and development of distinct ways to distinguish content management for different online courses. For example: OCW and MOOCs users have different learning needs and patterns, which might require that different instructional designs be applied. The re-use value of OCW materials should be 119 further developed, such as sorting and indexing the teaching materials of OCW. It can make users search and access the needed information easier.
As the number of courses grows, an index system needs to be established to assist users to search and find courses more efficiently. Other features, such as rating and review, can also assist users in choosing courses.
NTU can provide information or workshops promoting re-use or re-create learning materials on the basis of OCW content and materials. New ways of innovative teaching and learning also can improve OCW values, such as finding ways to assist teachers in using OCW materials for flip-teaching and blended learning.
Significantly, adding closed captions to courses for accessibility to all is on the list. Currently, only partial courses with closed captions provided by YouTube are available. This ensures that courses are accessible by users with special needs and helps reach a much wider audience. These myriad ways can increase the usage of open education resources for the benefit of learners, instructors, learning center personnel, and other interested parties. We are just beginning to experiment with and explore some of them.