An Exploratory Study of Effective Online Learning: Assessing Satisfaction Levels of Graduate Students of Mathematics Education Associated with Human and Design Factors of an Online Course

This exploratory research project investigated graduate students’ satisfaction levels with online learning associated with human (professor/instructor and instructional associate) and design factors (course structure and technical aspects) using a survey study. A total of 81 graduate students (master’s students who majored in math and science education) enrolled in an online math methods course (Conceptual Geometry) participated in this study. According to the results of this study, student satisfaction level is closely associated with clear guidelines on assignment, rubrics, and constructive feedback. In addition, student satisfaction level is related to professor’s (or course instructor’s) knowledge of materials.


Article abstract
This exploratory research project investigated graduate students' satisfaction levels with online learning associated with human (professor/instructor and instructional associate) and design factors (course structure and technical aspects) using a survey study. A total of 81 graduate students (master's students who majored in math and science education) enrolled in an online math methods course (Conceptual Geometry) participated in this study. According to the results of this study, student satisfaction level is closely associated with clear guidelines on assignment, rubrics, and constructive feedback. In addition, student satisfaction level is related to professor's (or course instructor's) knowledge of materials.
There has been clear evidence of rapid growth in online learning in U.S. higher education (Christensen, Horn, & Caldera, 2011;Nagel, 2010). According to Ambient Insight Research (2009), it is projected that more than 80% of all higher education students will take at least one online course by 2014. About 65% of higher education institutions in the U.S. have reported that they offer online education options (Allen & Seaman, 2005). Considering the current needs of online programs at the higher education level, the critical concern is how to maximize student learning by providing quality online learning experiences. Studies have shown that affective domains of students such as motivation and/or satisfaction are significantly related with their learning outcomes (Bryant et al., 2005;Eom, Wen, & Ahill, 2006). More importantly, when they are satisfied with online learning, students tend to be more motivated in learning and motivation leads to context (Graham & Scarborough, 2001). It is essential to closely investigate what elements and factors impact the level of student satisfaction. When students have a high level of satisfaction in learning, they obtain a high level of learning (e.g., Alavi, Wheeler, & Valacich, 1995). To provide students with an effective learning environment, Zhu (2012) recommends that student satisfaction levels should be taken into consideration in designing online courses and building online environments. Thus, it is important to investigate what factors are associated with student satisfaction level for online learning in order to maximize student learning. In this study, factors specific to an online mathematics methods course were investigated.
In this study, the researcher used a conceptual model based on the framework suggested by Piccoli, Ahmad, and Ives (2001) that both human and design factors influence student affective and cognitive domains and ultimately lead to effective learning.  In high-quality online courses, students learn as much as in face-to-face courses (Meyer, 2002). As Figure 1 presents, high-quality online courses are composed of high-quality  (Osmanoglue, Koc, & Isikal, 2013;Russell et al., 2009) and design factors (Nuangchalerm, Prachagool, & Sriputta, 2011). Empirical studies have shown how these factors impact student outcomes or adult learner outcomes. For example, Russell et al. (2009) investigated the effects of online courses for middle school algebra teachers and found significant impacts on teachers' mathematical understanding, pedagogical beliefs, and instructional practices associated with human factors, in this case math education instructors. When instructors provided teachers with a highly supportive environment, they tended to show positive outcomes in teaching and learning. O'Dwyer et al. (2007) emphasized quality communication as a way to build supportive virtual learning environments for students. They found the lack of communication between students and instructors affected student satisfaction levels in a negative manner. Osmanoglu et al. (2013) examined 19 pre-service and 7 in-service mathematics teachers on the use of online video case discussions and found that effective communications among peers and with instructors promoted pre-and in-service teachers' mathematical process skills (problem solving, connection, mathematical communication, reasoning and proof, and representation). In particular, the outcomes were shown to be more significant when the instructor participated in discussions.

Research Design
A cross-sectional study was conducted in which a survey was administered to participants one time. Cross-sectional study has been known as an effective method to provide a snapshot of the current behaviors, attitudes, and perspectives of participants (Gay, Mills, & Airasian, 2009). This method was used to measure what factors are associated with student satisfaction level in online learning.

Instrumentation and Pilot Study
For the purpose of this study, the researcher designed a questionnaire and conducted a pilot study in order to measure reliability of the survey instrument. The survey items comprise a total of 24 questions including four open-ended questions. The items were developed by reviewing online learning design-based research (DBR). DBR is an emerging paradigm which includes pedagogy and tools to help develop and sustain learning environments (Norton & Hathaway, 2008). The major principle of DBR is to produce the most effective learning environment by applying continuous cycles of design enactment, analysis, and redesign (Cobb, 2001). Following the principle of DBR, the survey measures human and design factors associated with student satisfaction levels. In this study, human factors involve professor and instructional associate (IA) or graduate assistant (GA). Design factors include course structure and technical aspects.
The survey consists of the following: five items to measure participants' thoughts on the professor, six items to measure participants' thoughts on the IA/GA, six items to measure participants' thoughts on course structure, and seven items to measure participants' thoughts on technical aspects. Twenty questions are answered on a sixpoint Likert scale ranging from 1 = strongly agree to 6 = strongly disagree, and four are open-ended questions (see Appendix for instrument).

Reliability.
According to Cox (1996), the term "reliability" is explained by the word "consistency." When an observed score is strongly consistent, the instrument has high reliability. To measure reliability of the instrument, a pilot study was conducted with 10 graduate students. The results were analyzed using Cronbach's alpha which is calculated, which is commonly used to test the reliability of questionnaire items (Cronk, 1999). In the pilot, only the 20 quantitative questions were used to calculate the reliability of the survey.
Cronbach's alpha on all 20 items was .92, which indicates very high reliability.

Program Overview
The program in which the participants were enrolled aims to provide a master's degree in Math and Science Education to in-service teachers or educators from pre-K through grade 12. Students who are admitted to the program take a total of 36 credit hours to complete the degree.

Online Course Structure and Instructional Design
The course (Conceptual Geometry) was offered once a year. Its curriculum focused on how to promote pedagogical content knowledge of geometric reasoning and conceptual understanding of geometry for teachers/educators from pre-k to grade 12. This course is required for all graduate students who are admitted to the program. It is five weeks long, with five modules (one module for each week). The instructor of the course utilized Blackboard (a web-based course management system). Table 1 presents the structure and instructional design of the course. In order for students to meet course expectations, they viewed video-taped lectures and PowerPoint presentations, completed required readings, took quizzes, completed assignment(s), and participated in theme-threaded forum discussions on a weekly basis.
Students were required to submit their assignments on Blackboard by midnight of every Sunday. IA/GAs were required to complete grading by every Wednesday and to monitor student discussions on a daily basis. The course instructor continuously monitored IA/GA work and graded student work if necessary.

Human Factors
Efficient online learning associated with professor. Table 2 shows how participants' satisfaction levels were associated with a professor/instructor. More than 95% of participants strongly agreed or agreed that their online learning was more satisfactory if their professors had the following characteristics and behaviors: knowledge of the course, prompt reply, constructive and timely feedback on student works. The highest percentage of participants (about 77%) strongly agreed that the professor's knowledge of course materials is important to make their online learning satisfactory.  SA-strongly agree, A-agree, N-neutral, D-disagree, SD-strongly disagree, NA-not applicable,

Missing-missing data (didn't answer)
One open-ended question asked participants about the qualities they expected for a course professor that would make online learning more satisfactory. Participants' feedback is categorized into the following four major themes: grading, knowledge, clear guideline/expectations, and availability/prompt reply. Illustrative examples of participant feedback reflecting these themes are presented below.

Grading
• I look for a professor to provide valuable feedback on assignments.
• I look for constructive feedback due to the need to learn what I can improve and what I am doing well on.
• I look for a professor that is consistent with grading and knows the course material.
• Someone gives assignments that help me grow as a teacher • When I receive an assignment back, I first look for the comments. I learn best by my mistakes.
• I also appreciate comments on my work I turn in so I know I am meeting the expectations of the professor and the course. Suggestions are great so I can better improve my work in the future as well.
• On time with grading and sticking to deadlines • Knowledge of current issues in math education • I also like to have a professor that is highly knowledgeable in the area of study presented in the course.
• Strong knowledge of the subject being taught • The professor should demonstrate knowledge about the course content and have the ability to present the content in a streamlined, intentional manner.

Clear Guidelines/Expectations
• Clarity of assignment details, due dates, grading scheme (rubrics) • I like instructors who understand the uniqueness of attending classes online. I prefer instructors who clearly state the assignment objectives and are readily available to answer questions.
• I look for professors that are very clear and concise in laying out their expectations for the course. I also look for clear and thorough descriptions of assignments. Most of all, I look for professors who have an equally high expectation of themselves as they have of their students.
• Clear when communicating (E-mails don't result in further questions from the student) • I look for clear guidelines of the teacher's expectations, i.e. rubrics, and try to follow those as closely as possible to achieve the highest mark that is indicative of my personal best.
• Clear, concise explanations of expectations Availability/Prompt Reply • Timely e-mail responses to questions • I expect the professor to be available for questions, organized, and prompt with replies.
• Particular qualities I look for in a professor of an online class include the professor being available in a timely manner if I have a question about something.   SA-strongly agree, A-agree, N-neutral, D-disagree, SD-strongly disagree, NA-not applicable,

Missing-missing data (didn't answer)
One open-ended question asked participants about the qualities they expected in an IA/GA to make online learning more satisfactory. Participants' feedback is categorized into the following three themes: grading along with feedback, accessibility/prompt response, and knowledge of course. Examples illustrating these themes are presented below. Grading along with Feedback • Grading in a timely manner with pertinent feedback • I look for constructive comments that can be used to improve the quality of my assignments when the IA is responsible for grading.
• Clear feedback on assignments, especially when the score is less than 100%.
Always provide reasons for deducting points (even if it's just 1).

Accessibility/Prompt Response
• I think that a quality that the IA should possess is accessibility. This person needs to be able to be reached at all times and expect that he/she will be contacted at any moment.
• Willing to answer questions, readily available • Someone who responds to you in a timely fashion and is helpful   • Clearly outlining assignments and due dates • Clear expectations, rubrics and examples of work provided by the professor are important so I am able to meet those expectations in my assignments.

Useful Teaching Resources
• …useful material available to benefit the students. I use the information to go back and refer to as a resource.
• A variety of meaningful learning tools and e-math materials allows me to learn more about the course content and be more successful in completing course work.
• I want the e-activities to be beneficial to me in this degree. I want to be able to learn something more than I already knew.

Module (Course) Organization
• Checklist with due dates is very beneficial. Weekly or either for the whole course where we can check each box when it is done.
• Courses should also have a clearly outlined pacing guide and due dates for each week presented at the beginning of the course.
• The syllabus and organization is essential. Each detail makes the course more effective and satisfactory. Helping us look ahead to future assignments is crucial for our success and maintaining our teaching jobs.
• I like to work at my own pace online. I like being able to also see what resources are online to help me through the course.
• Aligning the assignments with the objectives is a great way to make the course more effective.
Efficient online learning associated with technical aspects.  SA-strongly agree, A-agree, N-neutral, D-disagree, SD-strongly disagree, NA-not applicable,

Missing-missing data (didn't answer)
One open-ended question asked participants what technical aspects they expected to make their online learning more efficient and satisfactory. Participants' feedback is categorized into the following three themes: user-friendly, material availability/accessibility, and technical support. Participants' responses that illustrate these themes are presented below.

User-Friendly
• Online math materials are easy to navigate.
• Easy to follow prompts • Easy to use technology that is working effectively • Easy to use, easy to get help • I think that the course should be easy to navigate.
• User-friendly and very simple to navigate • Clear direction on how to use and access the material in the course is very important. Also, it is important that links to resources work. All Materials Available Online/Accessibility • The online course is great because everything that I needed was online (books, articles, videos, discussion). It really makes the learning less stressful.
• The course is more effective if all materials can be accessed online. Readings should be able to be accessed through the course site.
• Being able to access material without difficulty is most important.
• Working links to materials distributed electronically • I love that it is accessible anytime. It is great also to be able to see all of the modules to look ahead if needed. It is quite easy to access documents.
• Access 24/7 • Readily accessible materials and a delivery system with few glitches make the course more satisfactory.
• Being able to access the materials easily Technical Support • Tech support is imperative!
• The availability of technical support is very important, especially between 6pm-12am and weekends.

Conclusions
It is critically important for universities to understand the emerging online learning environment, particularly the level of student satisfaction which is closely associated with learning outcomes. The paradigm of adult learning has changed from contentcentered to learner-centered, which places more emphasis on learners than on course materials/contents (Magnussen, 2008). In this regard, this study focuses on learners and provides exploratory empirical data to contribute to the field of online math education in order to make online learning more meaningful to students.

Human Factors
Participants in this study thought that it was most important that a professor and IA/GA be knowledgeable about course materials and provide up-to-date effective strategies/pedagogical knowledge for teaching math. There was a slightly different expectation for instructional personnel: The highest percentage of students expected a course professor to be knowledgeable of course materials while they expected an IA/GA to be able to grade fairly with constructive feedback. Several participants stated that they needed constructive and clear feedback on their work as they attempted to improve their work as well as become a better teacher of mathematics. Prompt replies from the professor and IA/GA were also related to student satisfaction levels. This is a similar finding to that of Russell and her colleagues (2009) in their research conducted with teachers of middle school algebra. They found that instructors' and content experts' availability was a key component of student effective learning in online math methods courses. Availability of a professor (instructor, content expert, and/or facilitator) is especially important in online math education courses since the course work or assignments often require a clear understanding of the tasks in order to successfully perform and complete them. If a student does not correctly understand the task, it is problematic. The first step of math problem solving is "understanding a question/problem" (NCTM, 2011). Therefore, finding ways to maximize the availability of a course professor or facilitator is essential. For example, setting up online conferences (live chat, webinar) on a regular basis (e.g., weekly) and having virtual office hours during which students can communicate with the course instructor can facilitate easy access to instructors.

Design Factors
In terms of course structure, students considered clear assignment rubrics and guidelines to be important to make online learning satisfactory. Participants expressed strong feelings about receiving explicit course expectations and assignment guidelines/rubrics. Students tend to focus more on learning when online courses are planned with clear expectations and guidelines (Dykman & Davis, 2008;Ku et al., 2011;Moallem, 2003;Salmon, 2002). Clear explanation is a key for successful online learning because it helps to prevent misunderstanding of course materials and assignments.
NCTM (2001) strongly recommends that teachers of mathematics precisely present expectations of students as well as mathematical ideas in order to avoid student confusion in performing their tasks.
Participants also thought that a user-friendly system is an important factor of their satisfaction level. Not all students are proficient in using technology (Darrington, 2008), so it is important to provide a user-friendly system to students for efficient online learning. However, it is also necessary to provide adequate training (e.g., how to navigate course materials, how to upload and download materials, how to get technical assistance, etc.) in order to reduce the level of student frustration. This also prevents difficulties with online technology from interfering with learning (Cornelius & Glasgow, 2007).
This study found that students' perceptions of discussion boards were associated with the level of their satisfaction. This is an interesting finding that students thought that a discussion board was less significant to make online learning satisfactory. There have been many studies which showed the positive impact on learning of the use of online discussions as students share their own experiences, reflections, and insights from their life as well as their readings (Rettig, 2013). This study did not indicate whether the participants' experience with discussion boards in the current online course was negative or they were simply not fond of online discussions in general. Future researchers should further investigate the use of discussion boards for virtual discussions in terms of formats, structure, contents, and so forth. Finding effective, interactive, and collaborative ways to participate in discussions is an essential part of making online learning meaningful to students (Bell, Hudson, & Heinan, 2004;Russell et al., 2009).
Communication is "an essential part of mathematics and mathematics education" (NCTM, 2001, p. 60). Communication in math education is critically important since it promotes students' reasoning and proof abilities along with their collaborative skills as they share their own mathematical ideas and listen to their peers' perspectives. Further study is necessary to determine how to promote student participation in online discussions and communications with their peers (e.g., what helps students feel more motivated to participate in discussions/communications, what format helps students feel more comfortable sharing their mathematical thinking and insights about teaching, etc.). Gallie (2005) and Jung et al. (2002) suggest that instructor/professor involvement with online discussions is more effective than discussions held among students only.
Depending on who initiates the discussion, the outcomes of learning differ. According to Russell and her colleagues (2009), the accessibility of instructors and/or facilitators is even more key to engaging more students in virtual discussion in online math education than self-paced online discussions.
Assessing students online using tests (quiz, mid-exam, final exam, etc.) has been a challenge in higher education due to the lack of validity (Clarke, et. al., 2004;Hewson, 2012). Participants in this study indicated that an online quiz was one of the least effective factors among other design factors associated with student satisfaction levels.
This is a conflicting result based on previous studies (e.g., Dermo, 2009;Marriott, 2009) in which participants showed relatively positive and favorable attitudes toward online summative assessments. Thus, it is recommended for researchers to investigate both formative and summative assessment in online courses in order to adequately evaluate student learning and at the same time to successfully motivate student interest and increase their level of satisfaction. Online assessment needs to be re-conceptualized in order to promote higher order thinking levels. Work that is authentic and applicable to the real world is more meaningful (Dunlap, Sobel, & Sand, 2007;Fish & Wickersham, 2009). In addition, Hewson (2012) urges further studies to examine student affective domains regarding online assessments to determine the best methods to assess students in the way they favor and that lead to a low anxiety level.

Study Limitations
This study is limited in several aspects. First, more studies are necessary to investigate in more depth the human and design factors associated with student satisfaction levels and effective online learning. Especially, it is important to investigate how to make effective and meaningful online learning. For example, one might ask how to make course structure more meaningful and effective to meet the needs of students (i.e., does live instruction work better than videotape-based instructions?; does presentation work better than discussions or readings? etc.). Second, this study does not consider students' demographic information. Demographic variables should be taken into consideration to sensitively meet the needs of students. Third, rigorous study designs are necessary to investigate cause and effect relationships on individual factors (design and human factors) by applying a true experimental design. Individual factors can also be broken into several aspects to investigate how to make them effective for meaningful learning.
For example, design factors involve assessments, discussions, lecture types, live chatting, virtual conferences, and so forth. It is necessary to further investigate how each aspect impacts and/or is associated with student learning. Having the quiz available online is important. Using a discussion board is important. What makes your online learning more effective and more satisfactory in terms of course structure? Technical Aspects SA A N D SD NA Online distribution of learning materials is beneficial. Multimedia itself (e.g., audio, video) is useful. Using a discussion board is beneficial. Timely technical assistance/support is important. A user-friendly course delivery system is important. What makes your online course more effective and more satisfactory in terms of technical aspects? SA-strongly agree, A-agree, N-neutral, D-disagree; SD-strongly disagree, NA-not applicable