International Review of Research in Open and Distributed Learning Ukrainian E-Learning Platforms for Schools: Evaluation of Their Functionality

This article defines 27 criteria for evaluating the functionality of e-learning platforms, grouped into three macro groups: (a) learning management, (b) learning content management, and (c) communications and collaboration tools. The proposed criteria can be used to evaluate any e-learning platform’s functionality. They allow teachers and administrators to make conscious choices about the highest-quality e-learning platform


Article abstract
This article defines 27 criteria for evaluating the functionality of e-learning platforms, grouped into three macro groups: (a) learning management, (b) learning content management, and (c) communications and collaboration tools.The proposed criteria can be used to evaluate any e-learning platform's functionality.They allow teachers and administrators to make conscious choices about the highest-quality e-learning platform for their schools and developers to improve e-learning platforms' functionality.The developed criteria became the basis for rating the functionality of Ukrainian developers' eight e-learning platforms' and determining the degree of support (in whole or partly) of e-learning components, categorized on the cognitive, social constructivist, motivation, and e-learning theories (CT, SCT, MT, and E-LT).The results indicate that the lack of communication and collaboration tools necessary to ensure quality distance learning is the main problem of Ukrainian e-leaning platforms.Comparative analysis of the functionality of e-learning platforms and components categorized on the learning theories helped determine that only three of the eight Ukrainian e-learning platforms (Accent [Mobischool], Class Assessment, My Class) fully follow the CT, SCT, and MT, but these platforms are all commercial products; therefore, they only partially support the E-LT.Solving this problem will be facilitated by developing e-learning platforms with open access, financed by the state budget in the context of the development of open and distance learning for Ukrainian students, as well as improving communication and collaboration tools in the context of conforming e-learning components to the social constructivist learning theory.

Introduction
The COVID-19 pandemic has affected all educational institutions in the world.The government of Ukraine, trying to restrain the spread of COVID-19, decided to close all educational institutions on March 12, 2020.This lockdown lasted until the end of the school year (May 2020).According to UNESCO's global monitoring, this nationwide closure affected 1,676,550 primary school students and 2,376,878 secondary school students in Ukraine (UNESCO, 2020), all of whom massively moved to distance learning.
Distance education is defined by the Association for Educational Communications and Technology as "institution-based, formal education where the learning group is separated, and where interactive telecommunications systems are used to connect learners, resources and instructors" (Parchure, 2016, p. 63).The distance learning format has actualized teachers' use of e-learning platforms that differ both in structure and offered functions.Piotrowski (2010, p. 20) defines e-learning platforms as "software that provides the technical infrastructure on which e-learning activities can take place."Ouadoud et al. (2016b, p. 582) emphasize that "a type LMS (Learning Management System) e-learning platform is a software including services to assist teachers in the management of their course" Ecoutin (2000, p. 5) describes open and distance learning platforms as software that assists in distance learning and combines the tools needed "for the three main users-teacher, student, administrator." An important problem in the implementation of e-learning platforms in distance education is the lack of clear criteria for assessing their quality.In evaluating e-learning platforms, Tomczyk et al. (2020) looked at the following criteria in teachers' and students' surveys: general course quality, professionally prepared materials, content usefulness, visual design, and the innovative character of platforms.Pandu and Fajar (2019) and Abubakari et al. ( 2021) evaluated e-learning platforms via the User Experience Questionnaire (UEQ).The UEQ consists of six scales with 26 items reflecting the following basic components: (a) attractiveness, (b) dependability, (c) efficiency, (d) perspicuity, (e) novelty, and (f) stimulation (Abubakari et al., 2021, p 4;Schrepp, 2015).
However, surveys of students and teachers mostly provide their opinions about using e-learning platforms (Pandu & Fajar, 2019, p. 1) and are not sufficient in evaluating e-learning platforms' functionality and their ability to provide quality distance learning based on the e-learning components categorized by the following learning theories: cognitive theory (CT), social constructivist theory (SCT), motivation theory (MT), and e-learning theory (E-LT) (Kumar & Sharma, 2021;Schunk, 2020).
Defining criteria that can be used to rate the functionality of e-learning platforms, and to determine their effectiveness in the context of existing learning theories, will allow teachers and administrators to make a conscious choice about the highest-quality e-learning platform to use at their schools, and it will allow developers to see how to improve the functionality of their e-learning platforms.
The objectives of this study were twofold: 1. to define criteria for assessing the functionality of e-learning platforms; and 2. to perform a rating assessment of the functionality of Ukrainian developers' e-learning platforms and determine the degree of Ukrainian platforms' support (in whole or in part) of elearning components categorized by the CT, SCT, MT, and E-LT learning theories.Ouadoud et al. (2016a) developed the approach for the quality evaluation of e-learning platforms, which is based on "the quality model interactive systems" (Ouadoud et al., 2016a, p. 13) of International Organization for Standardization (ISO) standard number 25010: "The product quality model categorizes product quality properties into eight characteristics (functional suitability, reliability, performance efficiency, usability, security, compatibility, maintainability and portability).Each characteristic is composed of a set of related subcharacteristics" (ISO, 2011, s. 4.2).The researchers combined the characteristics presented in ISO standard 25010 into two categories-utility and usability-each of which was divided into subcategories (Ouadoud et al., 2016a, pp. 16-17, 19), which are presented in Tables 1 and 2.

Theoretical Framework for Evaluating E-Learning Platforms
Characteristics selected for evaluating an e-learning platform are developed via a software engineering approach with an emphasis on the technical aspects of the e-learning platform.In this study, we analyzed the quality of the e-learning platforms by one characteristic only: functional suitability.This helped us to determine how certain functionalities of a platform help with implementing e-learning components, defined according to different learning theories.
Table 1 summarizes the sub-characteristics of the functional suitability of e-learning platforms proposed by Ouadoud et al (2016a).The list of functional suitability sub-characteristics were modified and supplemented by functional characteristics, which, according to Colace et al. (2002), "must be absolutely present in an on-line learning platform."(Colace et al., 2002, p. 7) The following are the particular functional characteristics (Colace et al., 2002, pp. 6-7): • progress tracking, • multiple course management, • student groups' creation and management, • content inclusion in accordance with standards, • content importation, • new course creation in accordance with standards, • course importation from other producers, • reports on course frequency or use, • test creation, • course catalogue, • multiple-choice tests, • reports on test results, and • automatic evaluation of tests.
Important services that are necessary for efficiently training the authors of the mentioned research online include "textual or vocal chat, whiteboard, live video stream, virtual classroom, application and file sharing" (Colace et al., 2002, p. 5).Baggia et al. (2019, p. 53) combined various characteristics that "have to be considered when selecting the appropriate system for an individual case" into three main groups: 1 testing, e-commerce capabilities (the ability to sell online courses), and virtual classrooms.
3. Social learning and collaboration: This includes support social learning with collaboration features (live chat, blog modules, Web conference integration, following concept, content sharing and rating, discussion boards, file sharing, integration with social media networks, profiling and expertise capabilities, and gamification tools).
The works of Baggia et al. (2019), Colace et al. (2002), andOuadoud et al. (2016a) became the basis for developing criteria for assessing the functionality of e-learning platforms, presented in the results this study.Kumar and Sharma (2021, p. 11)   Our study of functionality of an e-learning platform in the context of their compliance of e-learning components, identified from the learning theories, will answer our main question: How well does an elearning platform help to achieve the goals of the learning process?

Methods
This study was conducted in four stages: pandemic-was added to our list of e-learning platforms.
Note that highly specialized platforms that are focused on in-depth study of a particular disciplinesuch as Lingva.Skills for the social project for learning foreign languages, Indigo Mental Training Club, and GIOS for learning mathematics-remained outside the scope of this study.

Stage 3
Developing the criteria for assessing e-learning platforms' functionality became the basis for qualitative analysis of e-learning platforms that were included in the list identified in the second stage.Each function was coded with one tag (+ or −) to remark on the presence or absence of a particular feature (i.e., the tag learners' management + is used to indicate the presence of the learners' management function).
The presence or absence of certain functionality of e-learning platforms was studied by qualitative analysis of information provided by developers on the sites of e-learning platforms (description of functionality, video presentation, etc.).
The final results consist of a set of evaluations composed of numerical ratings expressed in a range from 1 to 5, depending on the number of available functionalities in each of the three units.The maximum score a platform can receive is 15 points.

Stage 4
Comparative analysis of the functionality of e-learning platforms and e-learning components categorized on the learning allowed us to determine the degree of support (fully or partially) Ukrainian e-learning platforms of different e-learning components categorized on the CT, SCT, MT and E-LT.
Mathematical methods were used for processing the survey results, and graphical methods were used to construct diagrams and tables.

Criteria for Evaluating E-Learning Platforms' Functionality
The criteria proposed by Colace et al. (2002) 3).
A learning content management system includes "all the functions enabling creation, description, importation or exportation of contents as well as their reuse and sharing" (Colace et al., 2002, p. 2)."Set of Tools represents all the services that manage teaching processes and interactions among users" (Colace et al., 2002, p. 9).Whereas distance education technologies are divided into two modes of delivery, namely, synchronous learning (all participants are present at the same time) and asynchronous learning (participants access course materials flexibly on their own schedule) (Parchure, 2016), communication tools between teachers and students are divided into "two fundamental categories: asynchronous communication tools and synchronous communication tools" (Colace et al., 2002, p. 5).Therefore, the described approach made it possible to propose 27 criteria for evaluating the functionality of e-learning platforms, presented in Table 3.

Rating Assessment of Ukrainian E-Learning Platforms' Functionality and Determination of the Degree of Their Support of Different Learning Theories
Table 4 presents the e-learning platforms selected for assessment and created by Ukrainian developers over the last five years.The platforms are arranged chronologically depending on the year of development.Each platform is assigned a code number (Table 4), which presents the results of a qualitative analysis of the functionality of the selected e-learning platforms in accordance with the evaluation criteria proposed in Table 3.

Communication and collaboration tools
Asynchronous communication tools

Synchronous communication tools
Textual or voice chat The top five e-learning platforms in terms of functionality are Class Assessment, Accent (Mobischool), My Class, Classtime, and The Only School (Figure 1).

Ratings of Analyzed Platforms
The availability of communication and collaboration tools-such as whiteboards and virtual classrooms, which are necessary to ensure quality distance learning-is most problematic in these e-learning solving is one of the skills best promoted by modern e-learning" (Abubakari et al., 2021, p. 3).
An evaluation of the functionality of the e-learning platforms according to the criteria (Table 3) and the e-learning components categorized on the learning theories (Table 2) helped us determine how the analyzed platforms support the four learning theories (completely or partially).Only three of the analyzed commercial e-learning platforms' (Accent [Mobischool], Class Assessment, and My Class) support functions aimed at ensuring collaboration, interaction, extrinsic participation, cooperation, engagement, communication, constructive alignment, peer review, and evaluation/assessment/feedback-that is, they support all the components of SC theory: "The platforms that follow the Social Constructivist theory pedagogy will in turn deliver motivational and cognitive components" (Kumar & Sharma, 2021, p. 6).
The Accent (Mobischool), Class Assessment, and My Class e-learning platforms follow all three learning theory pedagogies completely.However, since these are commercial products, they only partially support the E-LT, which assumes increased accessibility and on-demand availability.
The Pidruchnyk.ua platform supports all learning theories only partially, as it provides access to electronic textbooks only and supports the functions of managing learning outcomes (electronic class register, electronic diary).The platform does not have tools for collaboration and group communication, for example.
The Euclid, Classtime, and The Only School platforms support all learning theories only partially as well, as they do not have sufficient collaboration, participation, and cooperation tools, and access requires payment.
The All-Ukrainian Online School platform, developed by order of the state, corresponds to E-LT, as it is free.However, it is an example of a mass open online course for middle and high school students, not a full-fledged e-learning platform, because it does not have many functionalities for learning management and content management, tools for communication between students and teachers, and teamwork organization.This platform therefore does not adhere the CT, SCT and MT: The educational content of the platform contains lessons in 18 main subjects: Ukrainian literature, Ukrainian language, Biology, Biology and Ecology, Geography, World History, History of Ukraine, Mathematics, Algebra, Geometry, Art, Basics of Law, Science, Physics, Chemistry, English and Foreign literature.Once launched, the content of the platform will be gradually supplemented according to the calendar plan.With the assistance of the International Renaissance Foundation, a mobile application of the All-Ukrainian School Online will be created soon and the functionality of the platform will be expanded, which will allow teachers to adapt teaching materials to the students' individual needs.(Ministry of Education and Science of Ukraine, 2020) A significant disadvantage of the All-Ukrainian Online School platform is not only limited functionality, in particular, the lack of communication and collaboration tools, but also the focus on middle and high school students only.To develop effective distance education in school, it is necessary to place on this platform all 23 electronic textbooks for students of the first, second, fifth, and sixth grades, developed by Ukrainian publishers in 2018-2019 and recommended by the Ministry of Education and Science of Ukraine for use in the educational process (Zhenchenko et al., 2020, p. 732).Assessing the functionality of e-learning platforms in the context of compliance e-learning components categorized on the learning theories (Kumar & Sharma, 2021) will allow developers to develop elearning platforms that follow all four learning theories (CT, SCT, MT, E-LT) completely.

Discussion
Various aspects of e-learning platforms' usability need further research via the UEQ (Pandu & Fajar, 2019, Abubakari et al., 2021) to improve teachers' and students' ability to use them.An in-depth assessment of student-teacher interaction through e-learning platforms can be based on Responsive Interactions for Learning (RIFL) measures-educator (RIFL-Ed) version (Rodrigues et al., 2021).

Conclusion and Implications
To assess the functionality of an e-learning platform, 27 criteria have been defined.They were grouped into three macro groups: (a) learning management, (b) learning content management, and (c) communications and collaboration tools.These criteria became the basis for rating assessment and determining the degree of support for various learning theories of the seven Ukrainian commercial

2
(a) searching scientific sources devoted to the problem of evaluating e-learning platforms, a literature analysis, and determining the criteria for evaluation of the functionality of e-learning platforms; (b) selecting e-learning platforms of Ukrainian developers for assessment; (c) rating assessment of the functionality of e-learning platforms of Ukrainian developers; and (d) evaluating Ukrainian e-learning platforms on a pedagogical approach and determining the degree of support (in whole or in part) of e-learning components, categorized by learning theories: CT, SCT, MT, and E-LT.At each stage, the research methods described below were used.Stage 1 At the first stage of the research, we searched the Scopus database for e-learning platforms' problems of quality assessment and their compliance with e-learning theories with the following queries: elearning platforms, evaluation e-learning platforms, and e-learning theories (only open-access journals were searched).The range of selected articles was expanded by analyzing references in the articles found with the specified search queries.Further study of scientific articles allowed us to identify several studies whose results became the basis for the development of criteria for assessing e-learning platforms' functionality-in particular, Baggia et al. (2019), Colace et al. (2002), and Ouadoud et al. (2016a, 2016b).Stage Computerization and informatization of the Ukrainian education system are accompanied by the development of electronic educational resources (EERs) and e-learning platforms.The commission on informatization of educational institutions was established by order of Ukraine's Ministry of Education and Science, Youth and Sports, dated November 25, 2011, No. 1364 (as amended by the order of the Ministry of Education and Science, Youth and Sports of Ukraine, dated November 29, 2012, No. 1341) for state examination, granting permission for the use of EERs and e-learning platforms in the educational process in all secondary schools of Ukraine.For the selection of e-learning platforms for evaluation, the protocols of the commission for 2016-2019 were analyzed, as the permission to use the platforms is granted for five years.Qualitative analysis of the protocols was based on the search phrases educational platform, educational system, and online platform, which allowed us to single out e-learning platforms among EERs of different types.Using an online information retrieval method, using the search query distance learning platforms and services, the All-Ukrainian School Online platform-developed in late 2020 with support from the Ukraine Ministry of Education and Science in response to distance learning challenges due to the COVID-19 , Ouadoud et al. (2016a), and Baggia et al. (2019) were summarized, clarified, and grouped into three categories: (a) learning management, (b) learning content management, and (c) communications and collaboration tools (Table other producers and plagiarism detection functions across almost all platforms.The latter function, plagiarism detection, is especially important given the problem of academic integrity in the educational environment.Only the Classtime platform has the anti-cheating function.All commercial platforms are recommended for use in the educational process marked without content.The developers of the platforms offer teachers and students their own (a set of test tasks, interactive exercises, theoretical materials on individual topics, etc.), but this content has not passed state examination.Only electronic versions of textbooks are recommended by the Ukraine Ministry of Education and Science on the Pidruchnyk.uaplatform, and the All-Ukrainian School Online platform hosts electronic courses certified by experts from the Ministry of Education and Science of Ukraine.
and collaboration tools platforms.The application sharing function, which is important during studying such subjects as computer science and technology, is not supported by any of the analyzed platforms.Only two analyzed platforms, Class Assessment and Classtime, support gamification tools.The first platform allows a creation of quizzes for students; the second allows a creation of team games and puzzles.Insufficient attention from developers of Ukrainian e-learning platforms to communication and collaboration tools negatively affects the quality of distance learning with these platforms: "Collaboration during problem- Ouadoud al. (2016aOuadoud al. ( , 2016b) ) studied and analyzed the evaluation dimensions of e-learning platforms relying on a software engineering approach based on the quality model interactive systems of ISO standard no.25010, which takes into account all technical aspects of interactive systems of e-learning platforms.In the context of our study, this model was used partly (functional suitability category).To evaluate e-learning platforms by criteria combined into the categories of performance efficiency, compatibility, security, maintainability, portability, and usability, more detailed technical information is needed.Colace et al. (2002, p. 7) distinguished, among the various functionalities of e-learning platforms, a representative number of the functionalities that must be absolutely present in any online e-learning platform.We accounted for the functionalities of e-learning platforms described byColace et al. (2002) during the development of the e-learning platforms evaluation criteria.Colace et al. (2002, p. 8)  consider that "in order to accurately evaluate the potentials of an online learning platform, it is important to pay attention to its three main components: Learning Management System; Learning Content Management System; Virtual environment for teaching and services associated with it."Baggia et al. (2019) divide the functional characteristics of e-learning platforms into three major groups: (a) learning content management, (b) course management, and (c) social learning and collaboration.With this in mind, we have grouped the evaluation criteria of e-learning platforms into three macro groups: (a) learning management, (b) learning content management, and (c) communications and collaboration tools.
emphasize that "e-learning components, identified from the learning theories are very much important for any platform.If these components are not integrated in the platforms, the success of learning cannot be guaranteed."In analyzing the theoretical perspective of e- connectivity/networking, increased accessibility, on-demand availability, interaction, participation, cooperation, collaboration, motivation, engaging, communication, intrinsic motivation, extrinsic motivation, intriguing ideas, self-determination, competence, autonomy, relatedness, cognitive effectiveness, convenient, reliability, efficiency, achievement, personalization, self-pacing, constructive alignment, higher learning outcomes, learner satisfaction, confidence, peer review, evaluation/assessment/feedback from instructors, improved tracking, flexibility, skills and knowledge improvement, and learner satisfaction.These characteristics can be further divided according to the four major learning theories (CT, SCT, MT, and E-LT)(Kumar & Sharma, 2021, p. 4), shown in Table2.

Table 3
Criteria of Evaluating E-Learning Platforms' Functionality

Table 4 E
-learning Platforms of Ukrainian DevelopersAnalysis results (Table5) show that most of the considered e-platforms have similar functionality for learning management and learning content management.There is a lack of course importation from

Table 5
Analysis of the Functionality of E-Learning Platforms