International Review of Research in Open and Distributed Learning

Volume 17, Number 1

January – 2016

Challenges of Transitioning to an e-learning System with Learning Objects Capabilities

Miroslava Raspopovic, Svetlana Cvetanovic, and Aleksandar Jankulovic

Belgrade Metropolitan University, Serbia

Abstract

Higher education institutions that have implemented blended and/or online learning need to be on the forefront of the cutting edge technological and educational advances in order to remain competitive and innovative. This task is usually very difficult due to the budget constraints that many institutions have. Keeping up with the current technological advances often brings not only financial challenges, but also transitional challenges that may put at risk learning quality and reputation of the institution, as well as performance of students. This work describes the features of the system, and the results and challenges of transitioning to an e-learning system that displays learning materials through a sequence of reusable learning objects (LOs) from the system that does not have these capabilities. Based on the research of existing methodologies and the results of this research, this work proposes a methodology for transferring into an LO-oriented e-learning system. Findings of this work reveal advantages, disadvantages, and potential obstacles of e-learning system implementation with LOs and give an overview of suggestions for implementation improvements. These suggestions are given based on evaluation of the implementation of an LO-oriented e-learning system, after the transition from the traditional non-LO oriented e-learning system.

Keywords: E-learning, distance learning, learning objects

Introduction

Rapid developments of information and communication technologies (ICT) are introducing new opportunities in the development of e-learning systems. Traditional e-learning systems are used to organize and publish learning materials, typically in the format of written lectures or video tutorials. However, the tendency for the new systems is leaning towards the personalization and interactivity of learning materials for each student (Chun, 2004). One of the ways to achieve this is by segmenting learning materials into smaller units, with the goal of being able to assemble lessons based on the already existing materials. In the context of e-learning materials, these smaller and reusable units are referred to as learning objects (LO). LOs can be used to guide students' learning by determining the sequence of LOs for each user based on their study curriculum and prerequisite knowledge (Cvetanovic & Raspopovic, 2012). LOs represent a good candidate in online learning for enhancement of learners' effectiveness, performance and experience. Usage of LOs and their repositories are said to offer many benefits to learners (AlMegren et al., 2013).

Much work has been done on analyzing how different types of e-learning systems influence learning experience and effectiveness. The focus of this paper is to identify potential problems and challenges in implementation of a new e-learning system with LOs when transitioning from a traditional e-learning system. The goal of this paper is to discover advantages, disadvantages, and potential obstacles of this transition and to propose suggestions for improvements. Transitional challenges that one institution may face are of interest, as this situation may pose difficulties that are not encountered when introducing the first e-learning system in the academic institution. It is considered important to integrate technology solutions in the process of teaching and learning and that the whole process should be appropriately designed (Cukusic et al., 2010).

According to literature research, much work has been done to propose different methods for implementing new information systems. While using a traditional Waterfall method can be time consuming and can put a big burden on the budget, new methods have been proposed to make this process more efficient. In recent decades, “agile” software development methodologies have been put forth as an alternative to traditional “waterfall” implementation; an “agile” implementation methodology has been put forth (Chan et al., 2009, Brhel et al., 2015).

Agile implementation focuses on short cycles (sprints), in order to make faster progress, and to improve monitoring the progress of the implementation. The idea is to complete one part of the system at the end of each system, and to allow re-evaluation of the project’s direction. This agile implementation methodology is usually used by SAP in the implementation of their Enterprise Resource Planning (ERP) packet systems. According to the SAP proposal (SAP AG, 2011), the implementation should start by identifying the most important functionalities (called “must have” functions) without which new systems cannot be put into operation. The next iteration allows for delivering the “nice to have” functions which are defined by the so called “delta list”. We adopted this agile approach and improved this agile process to be more suitable for the implementation of the e-learning systems.

The paper is organized as follows. Section 2 presents e-learning systems and their integration with learning objects, along with methods of evaluation of e-learning systems. Section 3 presents the research methodology used for result analysis, while Section 4 presents the results and discussion of this research. Section 5 concludes the paper.

E-learning systems and their integration with LOs

Progress in the field of e-learning has been very slow due to the problems that are mostly associated with the poor interface design of e-learning systems (Zaharias, 2005). On the other hand, research confirms the high importance of design and employment of methodologies, tools, and learning environments for stakeholders (both teachers and students), as well as usability evaluation of e-learning applications (Granic, 2008). A significant and important part of an e-learning system is the learning management system (LMS). LMS is the “front face” of the entire system and what users are able to see and interact with. An LMS is designed to organize and regulate the administrative tasks of schools and other educational organizations (Esther, 2008). According to Watson and Watson (2007), an LMS provides functionalities beyond instructional content such as management tracking, personalized instruction, and integration across the system. However, the LMS does not constitute the entire e-learning platform, especially when considering e-learning systems that use LOs. In addition to an LMS, an e-learning platform that can work with LOs has a system for creation and editing of LO content; it allows retrieval and reuse of LO, and delivery of appropriate LO to the LMS.

Not every LMS supports implementation and presentation of learning materials through a series of LOs. The problem with LO-oriented systems is a potential gap between designing different levels of learning and designing and planning a higher-level lesson or course. There are many examples that reflect the problems, from course design down to LOs design. The LDSE project provides the support for learning design including specific learning activities, lessons, and full courses. LDSE points out that designs at the ‘higher’ layer should incorporate ‘lower layer’ designs (LDSE, 2009). JISC D4L program (Beetham, 2008) set out the model to address linking between higher-level designs, which include course/lesson planning and LOs design. This model was represented as four layers: course design at the highest level, session or lessons planning, activity design (things learners actually do), and designing LOs. The key weakness of this model was the separation between layers of learning activity and learning objects that was treated as unsustainable.

In designing a learning object layer, two categories of design principles can be implemented: structural and pedagogical. The aim of structural principles is to create self-contained LOs with reusable content. Structural design principles include cohesion and de-coupling (Boyle, 2003). On the other hand, the aim of the pedagogical design principle is to create rich interactive learning experiences (Boyle, 2008). One system that graphically supports the design approach at the lesson/course level, and is inspired by IMS LD specification (IMS LD, 2003), is LAMS system (Dalziel, 2003). LAMS is based on the selection and sequencing of different pedagogical functions for presentation, discussion, and previously designed LOs.

For effective implementation of LOs within an e-learning system, it is recommended to have a set of assigned metadata for each LO. Metadata enable LO to be described, indexed, and searched. Several metadata standards have emerged and can be used to describe LOs and their resources: IEEE LOM, ARIADNE and IMS. However, the high quality design and development of LOs, enabled by pedagogical principles, is more important than the LO's metadata description. A new concept of Generative Learning Objects (GLO) was introduced with pedagogical principles in LO design (Boyle, 2006a; Boyle, 2008). GLO establishes an underlying learning design that can be created, inspected, and adapted to meet learners' needs. Learning design tools can be classified in four groups: authoring, ontology-based, standards-based, and generic form-based tools (Charlton, 2010).

There are many authoring tools that allow for the creation and reusability of learning design. Authoring tools can incorporate just structural, or a combination of both structural and pedagogical design principles. There are some authoring tools that should be mentioned such as Articulate Presenter, Snap, Raptivity, Xerte, Course Lab, Atutor, GLO Maker, and Microsoft content developer (LCDS). Generally, authoring tools should fulfill all features including pedagogy, reusability, and technology (Diwakar, 2013; Dag, 2014). For the purpose of this study, the DITA authoring tool was used.

Ontology enables the organization of learning materials around small pieces of semantically enriched LOs that can be easily organized into customized learning courses and delivered on demand to the user (Nejdl, 2001). It allows for associations between multiple LO repositories and solves the problem of metadata misconception. Ontologies are classified in three categories: (1) a content-domain ontology that solves the problem when semantically identical concepts are expressed by different keywords, (2) a content structure ontology that explicitly defines LO structures that may be useful in many cases when there is a need to reuse a specific part of an LO, rather than using the LO as a whole, and (3) context (didactic) ontologies formally specifying the educational/pedagogical role of a content unit (Stojanovic et al., 2001).

For the purpose of this study, IBM’s Darwin Information Typing Architecture (DITA) was implemented as a content structure ontology. DITA provides two structure components: topics and maps. A topic can be used to describe an overview, summary, or particular concepts or tasks as parts of a lecture, exercises, assignments, case studies, etc. The map is used to specify the chosen learning design model. In recent years, learning design has moved away from a sequential waterfall model, where output from one phase serves as an input to the next, and becomes a more fluid and nonlinear process that relies on frequent evaluation and alternation. In this view, learning design should be seen as a complex adaptive system where one component in the design process influences all other components, not simply a sequential element (Ritchie, 1999).

E-learning evaluation

E-learning evaluation is not a one-time activity but rather a complex process parallel to e-learning development as well as its implementation (Dvorackova & Kostolanyova, 2012). A large number of e-learning evaluation models exist. One of the most cited and implemented is the DeLone and McLean model (Raspopovic et al., 2014). The evaluation of an LMS must consider its usability and accessibility, as well as its didactic effectiveness in order to achieve educational goals. If an e-learning system is not usable enough, it obstructs students' learning and the learners would spend more time learning how to use the software rather than learning the contents (Wong, 2003). The usability of the system is of great importance for e-learning implementation, and therefore, it is also important for the transition from one system to another. It is imperative to research and implement an appropriate methodology for the system usability.

Several studies have investigated the usage of learning objects and their impact on improving the teaching and learning process (Boyle, 2003; Cochrane, 2005; Van Zele, 2003). Khor (2014) showed that users’ perception has a significant effect on the acceptance and adoption of usage of learning objects.

According to literature research, there are a number of methodologies for the evaluation of e-learning systems: the methodology for systematic usability evaluation – SUE (Ardito et al., 2006); the hierarchy structure for evaluating web-based e-learning systems – WELS (Shee & Wang, 2008); the hexagonal e-learning assessment model – HELAM (Ozkan & Koseler, 2009); the three stages model (Cukusic et al., 2010); and a model for using e-learning systems for sharing experiences, communicating with others, and getting feedback from student’s learning experiences (Wang & Chiu, 2011).

The reference model is the Layer 3 of the Institute of Electrical and Electronics Engineers (IEEE) Learning Technology Standards Committee (LTSC) - standard IEEE P1484.1 LTSA defines three components: Processes (boundaries, services, inputs and outputs of the learning system), Stores (repository of data that can be accessed by users), and Flows (connectivity and the type of information exchanged). The standard IEEE P1484.1 LTSA as a mode defined by an authoritative institution is proposed as a framework that encourages portability and reusability, and provides a good basis not only for architectural and functional design, but also for evaluation of candidate software products for a Virtual University Information System and e-learning platform (O’Droma et al., 2003).

International ISO standard ISO/IEC 19796-1, developed in consensus by the International Standard Organization (ISO), defines standards in learning, education, and training in general. The ISO/IEC 19796-1 standard was developed by the Working Group 5 “Quality Assurance and Descriptive Frameworks” of the standardization committee ISO/IEC JTC1 SC36. The quality standard contains the reference process model “Reference Framework for the Description of Quality Approaches” (RFDQ) to help stakeholders in learning, education, training, and especially in e-Learning or blended learning to document and (re-)define their everyday business and processes (ISO/IEC 19796-1:2005; Stracke & Hildebrandt, 2007). The Reference Process Model from the ISO quality standard RFDQ (ISO/IEC 19796-1) consists of seven process categories and 38 processes:

• Needs analysis (Initiation, Stakeholder Identification, Definition of objectives, Demand analysis);
• Framework analysis (Analysis of the external context, Analysis of staff resources, Analysis of target groups, Analysis of the institutional and organizational context, Time and budget planning, Environment analysis);
• Conception/design (Learning objectives, Concept for contents, Didactical concept/methods, Roles and activities, Organizational concept, Technical concept, Concept for media and interaction design, Media concept, Communication concept, Concept for tests and evaluation, Concept for maintenance);
• Development/production (Content realization, Design realization, Media realization, Technical realization, Maintenance);
• Implementation (Testing of learning resources, Adaptation of learning resources, Activation of learning resources, Organization of use, Technical infrastructure);
• Learning process (Administration, Activities, Review of competency levels);
• Evaluation/optimization (Planning, Realization, Analysis, Optimization/ Improvement).

In our research we analyze our suggested methodology for implementation of LO in e-learning systems by three components defined in the IEEE standard and through the seven process categories of EN ISO/IEC 19796-1. Analysis is conducted through a students’ survey.

Functionalities of e-learning systems with and without LOs

An e-learning system that does not have the capability to present learning materials through LOs (Type A systems) is designed as a system with the static and non-LO oriented content. Learning materials for Type A systems contain the materials that are presented and prepared in the same manner for all students. The objective of such system is usually to transfer the knowledge, information or skills from the teacher to students.

LO-oriented e-learning systems (Type B systems) are based on finely granular LOs. LOs represent modular and logically independent reusable units, even though they may be related to other LOs in a larger scope. As discussed by Sicilia & Garcia (2003), reusability is an essential and arguably the most important characteristic of the reusable LOs. Drucker (2000) compares the characteristics of non-LO oriented systems and LO systems through several dimensions: delivery, responsiveness, access, symmetry, authority, personalization, adaptability, etc. By analyzing these two types of systems, the author concludes that they show completely different characteristics.

According to the research of Baki and Cakirogly (2010) within learning objects in high school mathematics classrooms, the students’ views and approaches about the used objects have indicated that the innovations in courses by using LOs had a positive influence. Students have made positive comments about the motivational and learning themes, enhanced interactivity, and higher visual potentials. The negative views were generally about technological conditions and time-consuming, similar examples.

In the study of McCormick & Li (2005), teachers considered that LOs were beneficial tools for learning and that they could be used for scaffolding to understand concepts. Teachers were interested in using LOs in their classrooms. The preferences about teachers' LO use are consistent with the results of case study research about LO use in classroom environments (Ilomäki, Lakkala, & Paavola, 2006). The preferences are shaped through student, design, and content-based causes. The results about preferences indicate that organizing, structuring, and guiding processes of the course are crucial. In conclusion, as Kay & Knaack (2007) pointed out, students may benefit from LOs if they are comfortable with LOs offering good learning control, useful content, and clear instructions.

In order to describe differences between Type A and Type B systems, two categories of functionalities were considered: preparation and usage of learning materials. Functionalities and differences between these two types of systems are described in Table 1 and are mainly based on the literature review and previous experiences. Even though they support the same categories of functionalities, the teaching and learning scenarios in Type A and Type B systems are performed in different manners.

Table 1

Comparison of functionalities between Type A and Type B systems

Methodology for transitioning to a new e-learning system

When transitioning from one e-learning system to another, one should consider an appropriate transition model that will allow learners who are used to using a Type A system to maintain and/or increase effective learning. In our case, the goal was to transition from the system that views learning materials as a whole, to a system that allows sequencing of different learning objects (LO), along with different learning activities. As previously mentioned, in this paper the system that can work with LOs is referred to as an LO-oriented e-learning system (Type B), and those that cannot as non-LO oriented (Type A).

Implementation of the new system and the transition from Type A to Type B system can be done in different ways. Since implementing a new e-learning system using the Waterfall method can be time consuming and costly, in our approach we combined Waterfall and agile methods resulting into a hybrid model. The goal is to allow for the implementation of the new system to be structured, adaptable, and flexible.

Hybrid transitional model

Based on the Waterfall and agile methods, we suggested a Hybrid transitional model, as shown in Figure 1. In this model the first phase of the Waterfall method is kept, as it is necessary to plan organizational and system changes, which have to be in line with the institution’s goals and mission. In this model, we assume that learners' goals are set and are in alignment with institution’s goals. In other words, we assume that the course is already planned out, as it existed in the old system. If the course is new, the additional step of planning the course outline and learning objectives should be taken into consideration. Once the organizational and system changes are prepared, the backlog of “must have functions” are determined. These are the functions with highest priority, without which the new system cannot be put into operation and which are primarily related to publishing learning material in a Type B system based on learning material in the Type A system.

Figure 1. Using a hybrid model (Agile+Waterfall) for transitioning to a new e-learning system

“Must have” functions were determined based on the organization’s set goals, objectives, and needs. Choosing vital functionalities of the Type A system to be part of “must have” functions for a Type B system can be an option in order to allow students to transition and adjust easier with the least possible interruptions in the quality of their learning. As “must have” functions were mainly based on the previous experiences and literature review, it is the intent of this work to analyze whether these choices were good, and how they can be improved.

In the case of a Type B system, “must have” functions were chosen to allow for an adaptive model of learning design at lesson and course level, and an adaptive teaching/learning process performed through the iterative process.

The following functions were determined as “must have functions” for the Type B system:

• LO authoring - The aim of LO authoring is to create high-quality LOs based on both structural and pedagogical design. In order to create reusable LOs, cohesion and decoupling with agile methodology of LO development (Cvetanovic & Raspopovic, 2014) were used. Furthermore, to capture and organize LOs content, a DITA framework for structural writing was used. Since the LO structure and DITA topic structure are very similar, each LO was presented as a DITA topic that is defined as an aggregation of following components of an LO: title, prologue, content, related links, glossary, and learning sub-objects. Prologue contains information necessary for LO searching and retrieval such as categorization to which curriculum and academic program this LO was developed for, difficulty level of the learning content, and keywords. In order to provide prologue information, each LO is "enriched” with the metadata that describes what the learning content is about, according to a previously defined content ontology. Furthermore, LO content was organized into sections: each section has its own title and body which can contain paragraphs, images, figures, formulas, and links to external resources (PDFs, zips, multimedia files, programming source code, etc.). Keeping in mind that there are various authoring tools, for the purpose of this work, the QDITA authoring tool was used. QDITA is a customized authoring tool based on an open source DITA authoring tool (Guduric et al., 2013).
• Learning (instructional design) – For higher level designs, two types of maps were used: lesson maps and course maps. Lesson maps, in addition to different topic types (specialized for overviews, content, summary), include two types of activities: students’ self-assessment such as tests containing open questions, true/false, single select or multiple answer type questions, and activities that enable communication and interaction, for example, chat, wiki, forum, etc. In the initial phase of the project, basic functions of the old system were maintained and included in “must have” functions, which related to interactive questions in test format for self-assessment and discussion forums. The course map's purpose is to organize a sequence of learning materials, including LOs, weekly assessments (test, homework, exercises), and course projects. For lesson and course preparation, the LAMS system was used.
• Publishing – The publishing methods were chosen based on the previous learners’ habits. Output formats were mainly chosen to be published on web using XHTML, as well as in PDF format. LAMS can be automatically instantiated as a Web-based learning session using the LAMS delivery infrastructure.

Before learning material is developed for an individual course, it is recommended that the base course is developed. The base course will be used to train other instructors on how to adapt and enhance their course materials for the new system.

In order for an e-learning system with LOs to be successful, it is necessary to use high-quality LOs that are structured, adaptable, and ultimately reusable in different contexts. Even though one may try to maintain modularity and logical independence of each LO, in actuality, group of LOs are usually developed in order to meet the needs of a certain learner, usually having in mind a particular course. These LOs are developed by course instructors and/or multimedia developers (Boyle et al., 2006). While course instructors are mainly in charge of developing content and pedagogical methods for presenting the materials, the multimedia developer's role is to enhance LOs with visual presentation under the instructor's supervision. What should be kept in mind is that interaction between course instructors and multimedia developers in this model represents an iterative process. Once the course instructor develops the content for LO, this content is forwarded to the multimedia developer. Another step can be incorporated in this iterative process in which the learning content should be first forwarded to the reviewer, who should be a domain expert. Even though involvement of the reviewer can ensure and improve academic quality of LO’s content, at the same time it may increase the cost of the entire process. Once the instructor forwards the LO, the multimedia developer develops the multimedia based on the given specifications. In this iterative process, the resulting LO is given back to the instructor for a review. This iterative process between instructor, reviewer, and multimedia developer continues until all specifications and criteria are met, and the LO is published (Cvetanovic & Raspopovic, 2014).

Once LOs are published, the instructor can receive feedback from learners about the published material. More importantly, quality assurance and evaluation can be completed through involvement of the learner in the entire process, so that learning materials are tested with a certain group of students. This is referred to as “use before reuse” (Boyle et al., 2006). Considering that these LOs are used in a “live” scenario, these evaluations can be done from a one-week period to the entire semester. This feedback and analysis should give a better input about things that should be changed, adapted, and enhanced throughout the semester. More importantly, from this type of evaluation, conclusions can be made as to how the entire system can be improved. In this evaluation process, “nice to have” requirements and functionalities are determined. These are then used for the next phase of system improvements. “Nice to have functions” were determined based on the faculty and student evaluation throughout the semester, and finally at the end of the semester, after the evaluation of the entire system was completed.

Our proposed hybrid model elements are based on ISO dimensions (Table 2).

Table 2

ISO dimensions and hybrid model

Research Methodology

In this paper we propose a model for transitioning from a non-LO system (Type A) to an e-learning system that supports generation, sequencing, and publishing of LOs along with different activities (Type B). The impact on students’ learning and study process by transferring from a Type A to a Type B system was analyzed. The purpose of this study was to analyze effective ways of changing an e-learning system and how these changes influence satisfaction, learning experience, and usage of the system. Furthermore, it is of interest to analyze how this transition can contribute to student motivation and satisfaction. The examination of this evaluation was expected to reveal good practices, obstacles, and needs for future improvements. In the conducted case study at the academic institution, the Type A system was used for eight years, before deciding to change to a Type B e-learning system. During the first year of the transition to the Type B system, the process of transferring from the Type A to the Type B system was documented and an initial evaluation was performed. To illustrate the outcome of applying these practices, the illustrative case study examined the following:

• Students’ degree of satisfaction with the new system
• Student satisfaction with published materials
• Students’ motivation and driving forces to be active (i.e., communication with peers, interaction with the instructor, etc.)

Current literature and methods of developing, implementing, and evaluating information and e-learning systems were employed (Boyle et al., 2006). This resulted in adapting an agile methodology for the development of course content in form of LOs and overall implementation of the Type B system, presented in the following section.

Faculty was asked to develop their courses in the form of a sequence of LOs and interactive activities. Both online and traditional students were asked to evaluate these LOs by using these materials in their courses for the entire semester. Data was collected throughout the semester through student interviews and at the end of the semester through a web questionnaire. Data from student interviews was used to improve the quality of learning content throughout the semester. Statistics comparing students’ usage of the system and satisfaction with the new e-learning system was analyzed. 184 students participated in this evaluation, out of which 93 were traditional, and 91 online students. These results were compared to the previous year’s results, collected prior to the introduction of the Type B system. As a reference, the analyzed Type A system was an Oracle iLearn platform, and Type B was a Learning Activity Management System (LAMS) with DITA standard. It should be noted that even though illustrative examples are used by referencing certain e-learning systems and technologies used, it is not the intent of this work to focus on tools and technologies, but rather on the model for a systematic transition from a non-LO oriented system to an LO-oriented e-learning system. Based on the evaluation analysis, recommendations for improvements and future research are given.

Analysis of success factors of the implemented transition

The evaluation of the Type B system and its implementation was done continuously throughout the semester with traditional and online students. Traditional students used learning materials on the e-learning system in addition to traditional lectures and labs in the classroom in blended-style learning, while the online students used the same materials without attending the lectures. Each lesson on the e-learning system consisted of a sequence of different topics (LOs), interactive questions in test format, discussion forums, assessment tests, homework assignments, exercises, projects, and summarized learning materials in a written lecture form in PDF format. At the end of the semester, students were asked to give their opinion on their learning experience via a survey. Each survey question offered scaled marks from 1 to 5 (1- unsatisfactory, 2- sufficient, 3- average, 4- very good, 5- excellent ). 184 students participated in this evaluation; 93 were traditional, 91 were online students, while 108 were men and 78 women.

The student survey includes analysis of IEEE components Processes and Flows (survey questions for evaluation of students’ degree of satisfaction shown in Table 3) and Stores (survey questions for evaluation of degree of satisfaction about the quality of learning materials published on the Type B system shown in Table 6).

Based on the average marks on the survey of students’ satisfaction (Table 3), students were most positive about interaction and additional consults with instructors, 3.96 ±1.08, and their current learning motivation, 3.72 ±1.18. Similarly, they expressed their positive opinion about weekly course assignments, which helped them during their learning, 3.66 ±1.33 (with the median 4, which corresponds to a very good mark). Students expressed that the least contribution in helping them learn was provided by discussion forums on the Type B system, and that this type of activity did not contribute much to better understand the learning topic, 2.26 ±1.30, with the median of 2 (corresponding to a sufficient mark). These findings show that students’ learning experiences are centered on problem-based learning and communication mainly with course instructors, and also with their peers. These results derive “nice to have functions” that should be considered within developing and modifying LOs. These “nice to have” functions point out that pedagogical methods that focus on increasing efficacy in communication with an instructor and classmates should be integrated with a Type B system. One possible approach to increase communication is to incorporate a social learning aspect within the Type B system, and foster problem-based learning through teamwork within this environment, guided and administered by the course instructor.

Table 3

Survey questions for evaluation of students’ degree of satisfaction

Table 4

Result of survey questions based on the type of studying

*** - p <0.001, ** - p <0.01, * - p <0.05

Results from Table 4 were analyzed based on the variable “type of study” (online and traditional). It is clear that a statistically significant difference was established on most questions when two types of studies were examined. Traditional students were satisfied with established communication with classmates and they consider that this activity contributed the most in their learning process (P1_4 and P1_5, p <0.001). On the other hand, online students expressed the high importance of their interaction and additional consults with the instructor in their learning process (P1_10, p <0.01). At the same time, online students expressed the positive opinion about their activities on discussion forums, interactive activities, and exercises on the e-learning system (P1_8 and P1_9, p <0.001). Online students are also satisfied with the quality of online learning, and they think that their surrounding shares their opinion (P1_2 and P1_3, p <0.001). These results are in line with previously drawn conclusions about adding “nice to have” functions that will emphasize pedagogical methods that foster communication and problem-based learning, along with teamwork and group activities. One can notice that traditional students expressed lower satisfaction with the e-learning system. Lower satisfaction can be attributed to the fact that traditional students used the e-learning system in blended-style learning with the main way of learning being class lecture and the e-learning as an additional source (not the only source). For establishing the statistical significance a Mann-Whitney test was used.

Table 5

Correlation matrix (correlation coefficients) of marks given on the survey questions (P1)

*** - p <0.001, ** - p <0.01, * - p <0.05

It was of interest to analyze whether the correlation exists between the analyzed parameters in order to be able to better improve the system and predict future results of improvements. As shown in Table 5, it is evident that a statistically significant positive correlation exists between most questions. The degree of correlation was classified as follows: small (from 0.10 to 0.29), moderate (from 0.30 to 0.49), and high (from 0.50 to 1). These results show that students who are satisfied with discussion forums are also satisfied with interactive activities and exercises (P1_8 and P1_9, p <0.001); students who have a positive attitude towards the type B system consider that discussion forums (P1_7 and P1_8, p <0.001), interactive activities, and exercises (P1_7 and P1_9, p <0.001) are very helpful for better success in studying and knowledge acquisition; students who have established good contacts with their classmates also feel that this connection contributes to their better success (P1_4 and P1_5, p <0.001); students who have a positive opinion about the quality of online learning also consider that their surrounding have the same opinion (P1_2 and P1_3, p <0.001).

Table 6

Survey questions for evaluation of the degree of satisfaction about the quality of learning materials published on the Type B system

A second part of the survey attempted to analyze how satisfied students were with published learning materials (Table 6). Based on the average marks from this part of the survey, it is noticeable that the highest mark was given for written lectures in PDF format, which actually represent summarized materials and leave students without utilizing the full functionalities of the Type B system, 4.51 ±0.96, with median of 5 (corresponding to the highest mark). The lowest mark was given to discussion forums, 2.08 ±1.30, with median of 2. This result shows that students did not take in the new system fully, and that their tendency is to study based on the learning methods from the Type A system. This result is significant, as it should lead to the improvements and further development of interactive and self-assessment activities on the Type B system, which would emphasize more learning outcomes and significance of each LO and its activities in the students’ learning process. On the other hand, even though the discussion forum received low marks, this does not necessarily mean that the forum represents a poor “methodical tool”, but it could also signify that not enough students have used the forum, so the discussion never developed between students, as well as between students and teachers. This also leads to the need for further development of “nice to have“ characteristics of the LO system that should be implemented for further improvements by including more LAMS activities, and even be considered as “must have” in the next iteration. To step away from the old system, consideration should be given to remove residual functionalities of the Type A system because it detracts students from fully embracing the new system, such as written lectures in PDF format.

Comparison of the results between Type A and Type B systems

In order to compare students’ satisfaction with the Type B system, the results were compared with the analysis previously conducted for the Type A system. Students’ level of satisfaction was classified as satisfied (when given marks 4 or 5) and unsatisfied (when given marks lower than 4). It can be noticed that students’ satisfaction decreased after transitioning to the Type B system. Moreover, these results point out that introducing a new system with significantly different functionalities, such as a Type B system, can in the short term decrease students’ satisfaction with learning experiences (Table 7). When introducing a new system, it can be expected that during the adjustment period, students may show the preferences towards the old style of learning; hence, it is important to carefully introduce novelties and tend towards their improvements. This is the reason for introducing “must have” functions as functions that are closer to the Type A system, to allow students an adjustment period. However, results show that this approach did not allow students to embrace new functionalities fully. Future steps in determining “nice to have” functions should move away from Type A functionalities, so that pedagogical methods that increase efficacy of learning through communication and problems solving can be further explored.

Table 7

Comparison of students’ satisfaction before and after the transition

Conclusions

This paper proposed a hybrid transitional model for the implementation of an e-learning system that supports interactive learning through the sequencing of Los and interactive and self-assessment activities. The hybrid transitional model was based on the combination of waterfall and light agile methods. Content of learning materials from a non-LO oriented system served only as a baseline for developing content of LOs. The proposed hybrid transitional model required careful definition of the “must have” and “nice to have” functions. “Must have” functions have to be primarily based on the vital functionalities, without which the system cannot be put into operation. For the easier transition from the old to the new system, some of the “must have” functions were chosen based on the identified vital functions of the previous system. The main disadvantage of choosing functionalities of a non-LO oriented system to be a part of “must have” functions was instructors’ high level of familiarity with the previous system and the time and training needed for adoption of the new approach and structure of LOs. They in some way tried to maintain basic functions of the old system, which does not allow for full utilization of the new model. Furthermore, residuals from the new system also represented an obstacle among students, who were used to written lectures in PDF format and had a hard time accepting new structured, segmented, and reusable LOs.

Development of LO content, peer review of LO, and multimedia development were recognized in the proposed model as a part of the Development/production ISO dimension. A suggested methodology for implementation of LO was analyzed using three components defined in the IEEE standard and through seven process categories of EN ISO/IEC 19796-1. This evaluation was also tightly connected with an assessment of an LO e-learning system from the students' point of view, and according to students’ survey analysis, improvements of LO content with “nice to have” functions are determined. In order to test the proposed model, the evaluation of the hybrid transitional model was conducted on the case study based on the students’ survey in order to verify “must have” and “nice to have” functions, the success of the transition, and to point out the key points of the system that can be improved. Analysis showed that students’ learning experiences were centered on problem-based learning (weekly and project assignments) and communication mainly with course instructors, and also their peers. Traditional students gave high importance to establishing good communication with their classmates, while online students gave the same importance to their communication with the instructor and additional consults needed for their learning of the material. These findings point out that additional “nice to have” functions should be explored, in order to integrate pedagogical methods to suit the needs of the students. One approach to address this is to incorporate a social learning aspect within an LO-oriented system and foster problem-based learning and communication through teamwork and group activities, guided and administered by the course instructor.

For the future iterations of improvements of the system, “must have“ requirements should focus on development and implementation of more appropriate self-assessment activities, as students find them important. These activities are directly linked to how students perceive the importance of the e-learning system in their studying. Furthermore, future work should explore satisfaction and inputs of the system evaluation from the faculty and institution management. Moreover, it will also be of great benefit to compare students’ academic success and results from non-LO and LO-oriented systems. These analyses will be of interest, once the institution has gone through several iteration of improvements and implementations of “nice to have” functions.

Acknowledgment

The work presented here was supported by the Serbian Ministry of Education, Science and Technological Development (project III44006).

References

AlMegren, A., & Yassin, S. Z. (2013). Learning Object Repositories in e-Learning: Challenges for Learners in Saudi Arabia. The European Journal of Open, Distance and E-Learning, June, 26.

Ardito, C., Costabile, M. F., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (2006). An approach to usability evaluation of e-learning applications. Universal access in the information society, 4(3), 270-283.

Baki, A., & Çakıroğlu, Ü. (2010). Learning objects in high school mathematics classrooms: Implementation and evaluation. Computers & Education, 55(4), 1459-1469.

Beetham, H. (2008). Review: Design for learning programme phase 2. Retrieved from http://www.jisc.ac.uk/whatwedo/programmes/elearningpedagogy/designlearn.aspx

Boyle, T. (2003). Design principles for authoring dynamic, reusable learning objects. Australasian Journal of Educational Technology, 19(1), 46-58.

Boyle, T. (2006a). The Design and Development of Second Generation Learning Objects. In E. Pearson & P. Bohman (Eds.), Proceedings of World Conference on Educational Media and Technology 2006 (pp. 2-12).

Boyle, T., Cook, J., Windle, R., Wharrard, H., Leeder, D., & Alton, R. (2006). An Agile method for developing learning objects, Proceedings of the 23rd Annual ASCILITE Conference: Who’s Learning? Whose Technology? Sydney, Australia. December (pp. 91-99).

Brhel, M., Meth, H., Maedche, A., & Werder, K. (2015). Exploring principles of user-centered agile software development: A literature review. Information and Software Technology, 6 (2015), 163-181.

Chan, F. K., & Thong, J. Y. (2009). Acceptance of agile methodologies: A critical review and conceptual framework. Decision Support Systems, 46(4), 803-814.

Charlton, P., & Magoulas, G. A. (2010, July). Self-configurable framework for enabling context-aware learning design. In Intelligent Systems (IS), 2010 5th IEEE International Conference (pp. 1-6). IEEE.

Chun, A. H. W. (2004). The agile teaching/learning methodology and its e-learning platform. In Advances in Web-Based Learning–ICWL 2004 (pp. 11-18). Springer Berlin Heidelberg.

Cochrane, T. (2005). Interactive QuickTime: Developing and evaluating multimedia learning objects to enhance both face-to-face and distance e-learning environments. Interdisciplinary Journal of E-Learning and Learning Objects, 1(1), 33-54.

Cukusic, M., Alfirevic, N., Granic, A., & Garaca, Z. (2010). e-Learning process management and the e-learning performance: Results of a European empirical study. Computers & Education, 55(2), 554-565.

Cvetanovic, S. & Raspopovic, M. (2014). Development and Enhancements of Learning Objects for eLearning Systems Using Light Agile Method, Proceedings of the Fifth International Conference on e-Learning, Belgrade, Serbia, 13-17.

Cvetanovic, S. & Raspopovic, M. (2012). Design of Learning Object Ontology for the Database Course, Proceedings of the Third International Conference on e-Learning, Belgrade, Serbia, 108-113.

Diwakar, A., Patwardhan, M., & Murthy, S. (2012, July). Pedagogical analysis of content authoring tools for engineering curriculum. In Technology for Education (T4E), 2012 IEEE Fourth International Conference on (pp. 83-89). IEEE.

Drucker P. (2000). Need to Know: Integrating e-Learning with High Velocity Value Chains, A Delphi Group White Paper. Retrieved from http://www.delphigroup.com/pubs/whitepapers/20001213-e-learning-wp.pdf

Dag, F., Durdu, L., & Gerdan, S. (2014). Evaluation of Educational Authoring Tools for Teachers Stressing of Perceived Usability Features. Procedia-Social and Behavioral Sciences, 116, 888-901.

Dalziel, J. (2003). Implementing learning design: The learning activity management system (LAMS). Proceedings of the 20^th ASCILITE, 593-596.

Dvorackova M. & Kostolanyova K. (2012). Complex model of e-learning evaluation focusing on adaptive instruction, Social and Behavioral Sciences, 47, 1068-1076.

Esther, S. (2008). One-stop shopping with learning management system. T H E Journal, 35, 18-20.Granic, A. (2008). Experience with usability evaluation of e-learning systems. Universal Access in the Information Society, 7(4), 209-221.

Guduric, N., Jovev, Lj., & Nikolic, B. (2013, September). QDITA and LAMS integration. In Proceedings of Fourth Conference on e-Learning (pp. 121-125).

Ilomäki, L., Lakkala, M., & Paavola, S. (2006). Case studies of learning objects used in school settings. Learning, Media and Technology, 31(3), 249-267.

IMS Global Learning Consortium. (2003). IMS learning design specification.

ISO/IEC 19796-1:2005 (2005). Information Technology - Learning, Education, and Training — Quality Management, Assurance and Metrics — Part 1: General Approach. Geneva: International Organisation for Standardization (ISO).

Kay, R., & Knaack, L. (2007). Evaluating the use of learning objects for secondary school science. Journal of Computers in Mathematics and Science Teaching, 26(4), 261-289.

Khor, E. T. (2014). An analysis of ODL student perception and adoption behavior using the technology acceptance model. The International Review of Research in Open and Distributed Learning, 15(6).

LDSE. (2009). Learning Design Support Environment (LDSE). Retrieved from http://www.tel.ac.uk/ldse

McCormick, R., & Li, N. (2006). An evaluation of European learning objects in use. Learning, Media and Technology, 31(3), 213-231.

Nejdl, W. (2001). Learning repositories-technologies and context. In Proc. of ED-MEDIA 2001 World Conference on Educational Multimedia, Hypermedia & Telecommunications.

O'Droma, M. S., Ganchev, I., & McDonnell, F. (2003). Architectural and functional design and evaluation of e-learning VUIS based on the proposed IEEE LTSA reference model. The Internet and Higher Education, 6(3), 263-276.

Ozkan, S. & Koseler, R. (2009). Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation. Computers & Education, 53(4), 1285-1296.

Raspopovic, M., Jankulovic, A., Runic, J., & Lucic, V. (2014). Success factors for e-Learning in developing country: A case study of Serbia. The International Review of Research in Open and Distributed Learning, 15(3), 1-23.

Ritchie, D. & Earnest, J. (1999). The future of instructional design: Results of a Delphi study. Educational Technology, 39(1), 35-42.

SAP AG (2011) Agile: Implementing SAP Step by Step, Faster Results Increased Flexibility More Transparency.

Shee, D. Y., & Wang, Y. S. (2008). Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications. Computers & Education, 50(3), 894-905.

Sicilia, M. A. & Garcia, E. (2003). On the concepts of usability and reusability of learning objects. The International Review of Research in Open and Distributed Learning, 4(2).

Stojanovic, L., Staab, S., & Studer, R. (2001). eLearning based on the Semantic Web. In Proceedings of the World Conference on the WWW and the Internet (WebNet 2001), Orlando, FL, USA.

Stracke, C. M. & Hildebrandt, B. (2007). Quality Development and Quality Standards in e Learning: Adoption, Implementation, and Adaptation. In World Conference on Educational Multimedia, Hypermedia and Telecommunications, 2007(1), 4158-4165.

Van Zele, E., Vandaele, P., Bottledooren, D., & Lenaerts, J. (2003). Implementation and evaluation of a course concept based on reusable learning objects. Journal of Computing Research, 28(4), 355-372.

Wang, H. C., & Chiu, Y. F. (2011). Assessing e-learning 2.0 system success. Computers & Education, 57(2), 1790-1800.

Watson, W. R. & Watson, S. I. (2007). An argument for clarity: What are learning management systems, what are they not, and what should they become?, Teach trends, 51, 28-34.

Wong, B., Nguyen, T., Chang, E., & Jayaratna, N. (2003). Usability metrics for e-learning workshop on human computer interface for semantic web and web applications, 3-7 November, 2003, Catania, Sicily, Italy, Springer-Verlag, Berlin Heidelberg New York, LNCS No 2889, 235-252.

Zaharias, P. (2005). E-learning design quality: A holistic conceptual framework. In Encyclopedia of Distance Learning, vol. 2, Idea Group Inc, Hershey.

Challenges of Transitioning to an e-learning System with Learning Objects Capabilities by Miroslava Raspopovic, Svetlana Cvetanovic, Aleksandar Jankulovic is licensed under a Creative Commons Attribution 4.0 International License.