Developing, Using, and Interacting in the Flipped Learning Movement: Gaps among Subject Areas

The purpose of this paper is to investigate the current video collection of an open-access video website (TED-Ed). The research questions focus on its content as evidence of development, its viewership as evidence of use, and flipping as evidence of interaction in informal learning. In late September 2013, 686 video lessons were posted on the TED-Ed website that spanned 12 academic subject categories and 60 academic subject subcategories, as labeled and sorted on the TED-Ed website itself. The findings of the analysis of the TED-Ed video collection indicate several gaps in the humanities, social science, and natural science academic areas in terms of the number of video lessons and viewership. Despite the gaps in the numbers of video lessons and the viewership across those three academic areas, the areas have very similar averages of daily flipped lessons. The future research agenda should focus on the motivation of viewers to create flipped lessons as evidence of learning in an open learning environment.


Article abstract
The purpose of this paper is to investigate the current video collection of an open-access video website (TED-Ed). The research questions focus on its content as evidence of development, its viewership as evidence of use, and flipping as evidence of interaction in informal learning. In late September 2013, 686 video lessons were posted on the TED-Ed website that spanned 12 academic subject categories and 60 academic subject subcategories, as labeled and sorted on the TED-Ed website itself. The findings of the analysis of the TED-Ed video collection indicate several gaps in the humanities, social science, and natural science academic areas in terms of the number of video lessons and viewership. Despite the gaps in the numbers of video lessons and the viewership across those three academic areas, the areas have very similar averages of daily flipped lessons. The future research agenda should focus on the motivation of viewers to create flipped lessons as evidence of learning in an open learning environment.

Introduction
The "Flipped Learning" movement emerged in the early 21 st century from K-12 schools to higher education institutions due to the availability of information technologies and free software (Bishop & Verleger, 2013). Two popular terms, "Flipped Classroom" and "Flipped Learning" have been used in the literature. EDUCAUSE defined that " [t]he flipped classroom is a pedagogical model in which the typical lecture and homework elements of a course are reversed" (2012, p. 1).
However, the Flipped Learning Network (FLN, 2014, p. 1) clearly argued that those two terms are not "interchangeable." According to FLN, "Flipped Learning" must have "Flexible Environment," "Learning Culture," "Intentional Content," and "Professional Educator" (2014, p. 2). Bishop and Verleger (2013, p. 5) defined the flipped classroom "as an educational technique that consists of two parts: interactive group learning activities inside the classroom, and direct computer-based individual instruction outside the classroom." In addition to formal education institutions, a non-profit group named TED (Technology, Entertainment, Design) launched a website, TED-Ed, where videos from TED's recorded talks and originally produced videos are openly available. As of April 2014, TED-Ed's open access videos are classified into 12 academic disciplines. The flipping of TED-Ed videos by teachers for use in their students' learning activities results in the dissemination of those videos to a wider audience. Additionally, "flip" is also a reference to a nascent and evolving teaching method called "Flip Teaching" (TED-Ed, 2014).
Additionally, online information providers such as TED have offered an important research area for informal learning. Ziegler, Paulus, and Woodside (2014) pointed out that previous studies on informal learning often relied on learners' retrospectively self-reflective reports. With the popularity of various online learning communities (e.g., YouTube, TED), researchers are able to collect data from those communities. Most up-to-date studies on flipped learning were conducted in formal learning environments (Bishop & Verleger, 2013, p. 10). There is a lack of studies on informal learning.
TED-Ed was selected as the focus of our research as the "Flipped" movement emerged and evolved as key research in education. The purpose of this paper is to investigate TED-Ed's current video collections across all 12 disciplines as evidence of development, its viewing records as evidence of use, and its number of flipped lessons as evidence of interaction in informal learning.

Literature Review
Research on Flipped Learning Berrett (2012) reported general flipped learning development in U.S. higher education and pointed out that STEM (Science, Technology, Engineering and Mathematics) disciplines were the major curriculum targets. Bishop and Verleger (2013) analyzed 24 studies related to flipping learning. Twenty-three of the 24 studies were in higher-education settings with one in a high school. They pointed out that most literature regarding flipped classrooms is from academically oriented news articles and online blogs and suggested that future studies should be conducted in a controlled environment to examine students' learning outcomes objectively. Bishop and Verleger's (2013) recommendations reflected a preferred research approach based on residential campuses. However, as the flipped learning movement is also often present in the online open environment in addition to traditional residential campuses, Bishop and Verleger's research approach may not be suitable to online open learning environments. Additionally, Kim (2014) pointed out that a major challenge to the open online learning community is the creation of free and low-cost learning opportunities.

TED-Ed History
The TED-Ed website was created by TED in 2012 aiming to offer more interactive learning experiences to users when viewing a rich array of videos (DeSantis, 2012). According to the TED-Ed website: "TED-Ed's commitment to creating lessons worth sharing is an extension of TED's mission of spreading great ideas. Within TED-Ed's growing library of lessons, you will find carefully curated educational videos" (TED-Ed, 2014). In late September of 2013, 686 video lessons were posted on the TED-Ed website that spanned 12 academic disciplines and 60 academic sub-disciplines (Table 1). No fee or login is required to view any video lesson on the TED-Ed website. Any video lesson on the website can be customized for free, but TED-Ed does require the user to create a registration and to log in to the website before customizing a video. A customized video lesson is created by adding to or deleting from the following: 1. the text of the lesson's title; 2. prewritten text in various sections; 3. various sections themselves, such as "Think," "Dig Deeper," "Discuss," and "… And Finally;" and 4. prewritten questions ( Figure 1).
Also, users may add their own questions to the "Think" aspect of the lesson and exclude old discussions and include new discussions in the "Discuss" section of the lesson. To finish the creation of the customization, the user clicks a button titled "Finish Flip." In this research study, the terms "customization" and "flip" are interchanged. The availability of video lessons on the TED-Ed website for free viewing and customization allows the user to engage with the video collection for informal learning and flipped learning.

Online Informal Learning
The creation of the TED-Ed website is evidence of how emerging information and communication technologies (ICTs) have promoted the importance of informal learning (Cross, 2007;Downes, 2010). People must learn how to use new ICTs at work or in daily tasks because ICTs also serve as communication channels to offer instructional content. For example, Pierce and Fox (2012) asked pharmacy students to view video podcasts in a flipped topic module and those students' performance on the final examination significantly improved compared to performance of students the previous year who completed the same module in a traditional classroom setting. (2013)

Research Questions
To answer the research questions, we collected and analyzed data from the video lesson collection posted on the TED-Ed website. Data about the video lesson collection was collected and analyzed for the following three topics: 1. academic areas/disciplines/sub-disciplines and video sources as evidence of content development; 2. viewing records as evidence of use; and 3. number of flipped lessons (i.e. customization) as evidence of interaction in informal learning. The three topics for data collection and analysis address the following three research questions: 1. What is the content of the TED-Ed video lesson collection in terms of academic subjects?

What is the viewership of the TED-Ed video lesson collection?
3. What is the "flipping" pattern of the TED-Ed video lesson collection?

Research Methods
We collected the data from TED-Ed in late September, 2013. The following describes three different data collection procedures used to address our research questions.

Content and Sources of the TED-Ed Video Collection
We were interested in knowing how TED-Ed develops its video lessons to promote online informal learning. In order to address our first research question, we collected data in a hierarchical structure: three major academic areas (Humanities, Natural Sciences, and Social Sciences), 12 academic disciplines, and 60 sub-disciplines (Tables 1 and 2). The three-tier structure of academic area/ discipline/sub-discipline presents a view of how TED-Ed develops its video lessons. The rationale of the structure is to see any developmental gaps in both broad and narrow academic subject areas.  Figure 2 presents an example of the TED-Ed website's layout for an academic discipline. In this example, the academic discipline is "The Arts," which contains three academic sub-disciplines: 1.
"Visual Arts," 2. "Performing Arts," and 3. "Value of the Arts." Each of the twelve academic disciplines contains at least two sub-disciplines. Video lessons in each academic discipline or sub-discipline are displayed when selected by the user. In Figure 2, "The Arts" discipline is selected, so video lessons within that discipline are displayed on the right side of the webpage.
We used a spreadsheet program to record and calculate the distribution, viewership, and flipping of video lessons in each academic area, academic discipline, and academic sub-discipline. We recorded the total numbers of these three types of video lessons according to their academic subjects. The analysis of the types of video lessons revealed another dimension of TED-Ed's content.

Calculating Viewership of TED-Ed Video Lessons
We collected data on the number of views for a video lesson by reading the display of that video's image on the TED-Ed website (Figure 2). For example, the number of views is 301 for the first video lesson, "How playing an instrument benefits your brain." To obtain the publication date of a video lesson, we had to click on the video lesson's individual link to connect to the video's corresponding URL on YouTube's website. Figure 3 shows an example of a video lesson published on July 22, 2014. During the data collection period, we collected each video lesson's publication date and number of views.

Calculating the "Flipping" Pattern of TED-Ed Video Lessons
Data on the number of customizations for a TED-Ed video lesson was collected on its individual webpage by reading the number in the box that is located just to the right of the statement that reads: "Customize This Lesson" (Figure 1)

RQ #1: What is the content of the TED-Ed video lesson collection in terms of academic subjects?
Almost 50%  There are several significant gaps within the 12 academic disciplines. For example, over 55% of video lessons in the "The Arts" discipline are in the "Visual Arts" sub-discipline; over 54% of videos in the "Business & Economics" discipline are in the "Global Economics" sub-discipline; over 69% of videos in the "Design, Engineering, & Technology" discipline are in the "Technology" sub-discipline; over 80% of videos in the "Philosophy & Religion" discipline are in the "Religion" sub-discipline; over 30% of videos in the "Psychology" discipline are in the "Motivation and Emotion" sub-discipline; over 36% of videos in the "Science & Technology" discipline are in the "Life Sciences" sub-discipline; and over 40% of videos in the "Social Studies" discipline are in the "History" sub-discipline (refer to Appendix).

Developing, Using, and Interacting in the Flipped Learning Movement: Gaps among Subject Areas Chen and Summers
This work is licensed under a Creative Commons Attribution 4.0 International License.
52 Figure 4. The total video distribution by academic area. Figure 5. The total video distribution by academic discipline. Figure 6. The total video distribution by source type and academic discipline.

RQ #2: What is the viewership of the TED-Ed video lesson collection?
Even though the natural science area has the highest number of video lessons among the three academic areas (Figure 4), the humanities area has the highest daily average viewership per video (Table 3). "Mathematics" and "Literature & Language" are the two academic disciplines with the highest daily average viewership, followed by "Psychology" and "Science & Technology." The discipline of "Design, Engineering, & Technology" has the lowest daily average viewership ( Figure   7).
Original video lessons have higher daily average viewership among the 12 academic disciplines, except for "Design, Engineering, & Technology," "Mathematics," "Teaching & Education" (which has no original video lessons), and "Thinking & Learning" (Figure 8).

RQ #3: What is the "flipping" pattern of the TED-Ed video lesson collection?
Despite the differing numbers of video lessons and daily average viewership between the humanities, social sciences and natural sciences, the three academic areas have very similar averages of daily flips per video (Table 4). The "Literature & Language" discipline has the highest average (over 0.7 flips/day), followed by "Psychology" and "Teaching & Education" (both at 0.6 flips/day) and "Mathematics" (over 0.5 flips/day) (Figure 9).
In general, viewers flip more on original lessons ( Figure 10). Original video lessons in the "Health" discipline have the highest daily average of flips (1.1 flips/day), followed by "Literature & Language," and "Psychology" (both at 0.8 flips/day) and "Social Studies" (0.7 flips/day) ( Figure   10).

Discussion
We observe huge gaps among the academic disciplines in terms of distribution of content, viewership, and flipping.

Lesson Production Gaps
More video lessons for the STEM disciplines have been created by TED-Ed (i.e. "Originals").
However, lesson production gaps still exist within the STEM disciplines. For example, the "Life Sciences" academic sub-discipline constitutes more than 36% of the 181 videos in the "Science & Technology" academic discipline, which also consists of the following sub-disciplines: "Earth & Space Science" (19.3%), "Physical Science" (24.9%), "Environmental Science" (14.9%), and "Nature of Science" (4.4%) (refer to Appendix). Another example is the "Technology" subdiscipline, which constitutes 69.4% of all videos in the "Design, Engineering, & Technology" discipline (refer to Appendix). These gaps indicate that a balanced production approach is 58 needed for the future development of video lessons on the TED-Ed website. Further research could provide data about the availability to TED-Ed of content for video lessons in various academic disciplines and sub-disciplines. Additionally, the use of "Technology" in one discipline and one sub-discipline shows that a review of the categorization of the academic disciplines is also needed.

Uneven Viewership
The number of video lessons in an academic discipline does not necessarily correlate to daily average viewership. For example, the "Mathematics" discipline has the highest average daily viewership (Figure 7), but it ranked eighth among the twelve disciplines in terms of the total number of videos ( Figure 5). Another example is the "Science & Technology" discipline, which contains 181 videos (the highest total number of videos for a discipline) ( Figure 5), yet it ranked fourth among the twelve disciplines in daily average viewership (Figure 7). Additionally, originally produced video lessons tend to get higher daily average viewership (Figure 8

Even Flipping of Lessons
Another important finding is that the humanities, social sciences, and natural sciences have very similar daily averages of flipped lessons per video (Table 3)

Research Limitations
There are several limitations to this study: • Only external observations are reported and interpreted: Reporting and interpretation are based on our own external observations of available video lessons, viewership and flipping of lessons. The factor of website downtime was not included in this study.
Future research should extend to internal data from TED-Ed to reveal more insight on the development of informal open learning.
• Desirable internal data: Internal data from TED-Ed could be its lesson development strategies or policies, production procedures, web traffic logs, and user profiles. By combining the external and internal data sets, we would be able to study additional factors influencing lesson development, viewership and flipping behaviors.
• Study period: Despite the dynamic nature of web traffic, we only collected the external data in a six-day period. The short time span limited the depth of the research findings.
• Measuring matrix for informal online learning: The open learning movement is in its beginning stages and it lacks a comprehensive measuring matrix. The rapid development of web-based technologies and concepts also make it difficult to develop appropriate matrixes for assessment.
However, the listed limitations also provide research opportunities for future studies.

Conclusion
In sum, our analysis of the TED-Ed video lesson collection presents evidence for desirable changes in terms of future development at TED-Ed. Significant gaps were observed in available content across academic disciplines. As TED-Ed's mission is to support teachers and to inspire the curiosity of learners, a more balanced offering of video content by discipline is desired. TED-Ed should review its production approach and create more video lessons for the academic disciplines with less content.
Learning assessment is also an important issue in the flipped learning movement. Policy makers, designers, and instructors have been seeking best-practice strategies to collect evidence to assess the impact of flipped learning on learners' experiences and outcomes. Although TED-Ed is a free content provider, assessment of its impact is still important in fulfilling its mission. In this study, we focused on the content, viewership and customization behaviors as forms of assessment.
Additionally, TED-Ed needs to monitor the discrepancies of viewership among the academic