Quality Assurance for Online Higher Education Programmes: Design and Validation of an Integrative Assessment Model Applicable to Spanish Universities

The quality assurance of online Higher Education online programmes is one of the great challenges faced by Spanish universities. Regular assessment of these programmes is essential in order to take actions to improve their quality. The said assessment should be complex and include all of the components of the programme, as well as its planning and implementation stages and its effects. The purpose of this paper is to present a model designed to assess the quality of online Higher Education online programmes that includes the assessment of the quality of the programme itself, as well as its continuous assessment. In order to design the model, the author conducted a bibliographical analysis of different standards, models, and guides developed in Spain and other countries to assess online education. The model was validated by 23 international online education experts. The results of the validation were triangulated with specialized literature, thus allowing the author to make decisions regarding whether to change the model by keeping, reformulating, or removing a dimension or indicator. As a result, two variables, fourteen dimensions, and 81 indicators were obtained. In order to verify the utility of the model it was applied in the assessment of four online programmes. The model guides the persons in charge of the implementation of online programmes and allows to conduct a more comprehensive assessment of the programme in order to discover its strengths and weaknesses, and opportunities for its improvement. The model can be also applied by online programme designers as a guideline for creating other, high quality programmes.


Résumé de l'article
The quality assurance of online Higher Education online programmes is one of the great challenges faced by Spanish universities. Regular assessment of these programmes is essential in order to take actions to improve their quality. The said assessment should be complex and include all of the components of the programme, as well as its planning and implementation stages and its effects. The purpose of this paper is to present a model designed to assess the quality of online Higher Education online programmes that includes the assessment of the quality of the programme itself, as well as its continuous assessment. In order to design the model, the author conducted a bibliographical analysis of different standards, models, and guides developed in Spain and other countries to assess online education. The model was validated by 23 international online education experts. The results of the validation were triangulated with specialized literature, thus allowing the author to make decisions regarding whether to change the model by keeping, reformulating, or removing a dimension or indicator. As a result, two variables, fourteen dimensions, and 81 indicators were obtained. In order to verify the utility of the model it was applied in the assessment of four online programmes. The model guides the persons in charge of the implementation of online programmes and allows to conduct a more comprehensive assessment of the programme in order to discover its strengths and weaknesses, and opportunities for its improvement. The model can be also applied by online programme designers as a guideline for creating other, high quality programmes.

Introduction
According to figures provided by the Spanish Ministry of Education, Culture, and Sports (2016) more than 15% of Spanish students study online. Given this high percentage, guaranteeing a high quality of online education is one of the main objectives of Spanish universities that offer online education programmes. This way, these universities seek to put an end to the assumption that the quality of online education cannot be as high as that of traditional education.
Furthermore, there is a discussion among Spanish academic groups regarding the appropriate method for assessing the quality of online education programmes with the purpose of answering the question of whether this quality should be assessed based on the same criteria as those applied to traditional programmes. The dominant view is that the assessment of the quality of these two types of programmes-online and traditional-is basically the same and only differs when it comes to the meaning given to the criteria and the instruments used to make measurements (Padilla, 2005;Marúm-Espinosa, 2011). As a result of this approach, the current trend in assessing online education programmes, especially when it comes to universities that offer both traditional and online education, is to assess them in the same manner as the traditional ones. Thus, online education programmes are assessed by using the criteria and indicators designed for assessing the quality of traditional education programmes without applying quality dimensions and indicators suitable to the context of online education. Moreover, Spanish accreditation organizations also assess and certify online programmes by applying the same models as the ones applied to traditional education programmes (Chmielewski, 2013).
Assessing online programmes in this way is a major drawback, as online education differs from traditional education when it comes to its organization and functioning. Online education is not equivalent to traditional models, and thus the same quality assessment mechanisms and models should not be applied to both types of education (Jung & Latchem, 2012;Veytia & Chao González, 2013).
Some models exist that seek to provide a response to the issue of quality assessment in online Higher Education programmes in Spain, such as those developed by AENOR (2012), ANECA (2012), AQU (2007). These models, however, combine a variety of approaches and sometimes respond to contradictory paradigms. They thus use divergent dimensions (and assign different meanings to these dimensions) to assess the quality of online education. Moreover, the indicators proposed by these models rarely highlight the need to assess all pedagogical and technological elements of the programm, its planning and implementation stages, and its impact, as postulated by Martínez Mediano (2013): Programme evaluation is the systematic collection of information regarding a programme in order to meet specific needs, that is focused on 1) the quality of the programme itself, its basic elements, structure and coherence; 2) the planning of its launching, taking into consideration human, material and organizational resources, 3) the development of the programme and 4) the results of the programme. (p. 197) The lack of knowledge in Spanish universities when it comes to the correct assessment of online education programmes, and the lack of models that could contribute to said assessment, inspired us to conduct our own research. The general purpose of the research was to design an applicative model for assessing online Higher Education online programmes in Spain by integrating the assessment of the quality of the programme itself and the ongoing assessment of the programme. This paper presents, 128 from a panoramic point of view, the process of designing and validating this model, which is expected to become a useful tool for assessing and improving all of the components of education online programmes as well as the three stages of their existence-that is, the initial stage, the development stage and the final stage. It should be noted that this global approach has not been addressed so far (or has only been partially addressed), as has been noted by authors such as Butcher &Hoosen (2014), andChmielewski (2013).

Assessment of the Quality of Online Education Programmes
An online education programme is a document (computerized or not) that covers actions designed to achieve the educational objectives for a determined period of time and which, in turn, is composed of a set of elements necessary for the development of a learning/teaching process regarding a determined subject taught online. It is conducted in its entirety using communication and information technology with the technological support of an online platform and the support of online teachers who assist and guide the students' academic development through different interactive telecommunication systems (Marciniak, 2016).
It seems that there is not much point in separating the assessment of the components of the programme-the traditional purpose of assessment-from the ongoing assessment of the programme. These are two sides of the same coin, if we take into account that the assessment of an online programme-considered as an organizational scheme of pedagogical and technological componentsonly makes sense if we assess the quality of those components together with the implementation of the programme and the results obtained from its implementation.
As a result, except for the evaluative research and other purposes, it is perfectly possible to harmoniously integrate the assessment of the quality of the components of the programme (i.e., the assessment of the quality of the programme itself) and the ongoing assessment of the programme. In this way, the assessment will be complete, will allow for an in-depth diagnosis of the current quality of the programme and will allow to reflect on what needs to be modified in order to ensure the highest quality possible. Furthermore, it will make it possible to determine whether the programme has been designed, developed, and implemented in a way that will produce the desired effects.
Assessment of the quality of the online education programme itself. The assessment of the quality of the programme itself is defined as the assessment of all of the elements that constitute an online programme (Marciniak, 2016).
Based on a bibliographical review (Barnard & Echols, 2015;Berge, Collins & Dougherty, 2000;Higher Education Compliance and Quality [HECQ], 1997;Meza, 2012;Rushby & Surry, 2016;Shattuck, 2014, among others) it has been established that there are no formulas or schemes defining the components that should be included in online learning programmes or in the assessment of such programmes. This is due to the fact that each programme is unique and, thus, its structure and elements depend on its objectives, contents, resources, and so on. Nevertheless, the Higher Education Committee on Quality (1997) proposes the following elements that should be taken into account while designing and assessing online programmes: learning objectives, teaching strategies, learning assessment strategies,  Online programme justification (What is the reason for the existence of the online education programme?)  Online programme objectives (What is the online education programme for?)  Student profile (Who is the online education programme for?)  Thematic contents (What is going to be taught?)  Learning activities (How is the online education programme going to be carried out?)  Online teacher profile (Who is going to conduct the online education programme?)  Didactic materials and resources (What is the online education programme going to be carried out with?)  Learning assessment strategies (How is the student's learning process going to be assessed?  Tutorial (What support is going to be offered to the students during the learning process?)  Virtual classroom of the programme (What is the virtual environment of the programme going to be like?) These components define the quality of online Higher Education programmes and, thus, should be assessed in order to identify the strengths and weaknesses, and the opportunities for improvement, of each programme.
Ongoing assessment of the online education programme. The ongoing assessment of the programme provides constant information regarding its progress. This information allows for the improvement the programme through feedback.
Pérez Juste (2014) and Surry and Ensminger (2009) affirm that the assessment of an education programme (whether traditional or online) should be designed to be carried out in three stages (initial, development, and final). Furthermore, each stage should have its own entity and objectives and the three stages combined should contribute to the improvement of the programme as a whole.

130
The initial assessment of an online programme is carried out before the programme is launched and has two main purposes: (1) to launch the programme under favourable conditions and (2) to establish the level of readiness of the programme for its launch so as to minimize the risk of failure.
The processual assessment of the programme evaluates the performance of the programme in its development stage and the possibilities for its improvement. This assessment has two main purposes: (1) To facilitate the timely making of immediate decisions regarding the improvement of the programme based on the results of the assessment and (2) to collect data for subsequent decisions (Pérez Juste, 2014).
The final assessment of the programme is focused on the achievements made by the programme. The purpose of this assessment is to verify whether the objectives of the programme have been reached, measure its effects, and assess student satisfaction with the programme.

Referents for Designing the Model
The specific nature of online education makes it necessary to draft, develop, and assess a series of educational actions for its correct development, and to determine unified guidelines in order to guarantee that these actions lead to high-quality results. To address these challenges, several authors have developed initiatives (standards, models, and guides) related to the quality assurance of online education. These initiatives can be used by universities as allies in their assessment of educational programmes. Some of the aforementioned initiatives are internationally recognised, while others have been designed to be applied in Spain.
International referents. It is worthwhile to consider the five pillars of quality online education model developed by the Online Learning Consortium (OLC, 2002) in the United States. The purpose of this model is to assist institutions in the identification of objectives related to online education and to measure their progress. This purpose can be fulfilled by assessing five pillars that constitute the quality of online education: (1) learning effectiveness; (2) faculty satisfaction; (3) student satisfaction; (4) scale; and (5) access.
In Latin America, the Latin American and Caribbean Institute for the Quality of Online Higher Education (CALED, 2010) developed the Guía de autoevaluación para programas de pregrado a distancia (guide for the self-assessment of distance undergraduate programmes) in order to improve the quality of the distance Higher Education offered by institutions in Latin America and the Caribbean. The guide includes a series of self-assessment guidelines within the certification process offered by CALED. The following areas of online education are assessed in this self-assessment: technology, training, instructional design, student information, and services.
While describing initiatives related to the assessment of the quality of online education, we must also mention the European Foundation for Management Development's (EFMD, 2006) technologyenhanced learning accreditation (EFMD CEL) programme. EFMD CEL is an accreditation process that assesses online education programmes with the purpose of improving the quality of online education programmes offered worldwide. The core of this process is the self-assessment of the programme by a group of internal auditors who assess the programme based on: programme profile, pedagogy, economy, technology, organization, and culture.

131
The aforementioned programmes are not the only programmes developed to assess the quality of online education. Table 1 contains the comparative analysis of some of the models analysed during the bibliographical research. The analysis has been arranged chronologically. Table 1 Comparative Analysis of Some of the Models Developed to Assess the Quality of Online Education

Model characteristics Assessment dimensions
The e-learning maturity model (eMM ;Marshall, 2006) This model constitutes the framework for the improvement of the quality of online education programmes. It has been developed to guide organisations in their understanding of their capacities regarding e-learning, and to help them in the assessment of its quality.  (Raeesi, Qorbani, & Akhgar, 2010) This model offers a series of indicators to assess the knowledge sharing process carried out through online education programmes. These indicators are divided in two groups: (1) indicators of quality measurement of e-learning with respect to knowledge sharing; and (2) indicators of knowledge sharing that directly affect the e-learning process.
Each group of indicators is assessed based on three perspectives: •Individual •Organisational •Technical 132 PDPP model (planning evaluation, development, process evaluation, and product evaluation; Zhang & Jiang, 2012) The PDPP Model is based on the Stufflebeam assessment model and on the characteristics of online programmes. It proposes a system for assessing the quality of these programmes based on four dimensions and 24 indicators.
• Planning evaluation • Development evaluation

• Process evaluation •Product evaluation
Conceptual framework for quality of e-learning (Ossiannilsson & Landgren, 2012) The goal of the model is to guide institutions in the development, implementation, evaluation, and internalization of e-learning in Higher Education. •Accessibility •Flexibility •Transparency •Interactiveness •Personalization •Participation

•Productivity
A model for the online education quality assessment (Giorgetti, Romero, & Vera, 2013) This model proposes a series of dimensions and criteria to assess the educational quality in order to continuously improve it.
•Professional training •Institutional and administrative management •Support for professional training A model and an index for elearning quality assessment (Petkova & Radeva, 2014) This model is focused on the measures for the generalized quantitative assessment of the quality of distance learning platforms. It offers a geometric index to calculate different factors (characteristics) and subfactors, and to complete a final, generalised assessment of the quality of e-learning. •Interaction •Staff support •Institutional quality •Assurance mechanism •Institutional credibility •Learner support •Information and publicity and learning tasks These analysed models-as well as other models that could have been quoted, the utility of which is undeniable-enable the assessment of online education programmes through a series of dimensions that contain the information necessary to draft assessment plans. The main problem is that the variables and indicators proposed by these models rarely highlight the need to assess the quality of the programme, as well as the quality of its planning, implementation, and results.
Another weakness of the analysed models is the lack of consensus regarding the number of dimensions. Some models propose the assessment of only three dimensions, while other models propose to assess up to eight dimensions. It is further observed that we do not encounter any dimension common to all analysed models. Another weakness of the aforementioned models lies in the differing meaning given to the dimensions, as their interpretation differs depending on the author and the methodology.

Spanish referents.
In Spain, few authors have developed initiatives (standards, models, guides) related to guaranteeing the quality of online education that can be used by Spanish universities as assessment allies as they move forward the assessment of educational programmes. Among others, we should mention the Association for Standardisation and Certification of Spain's (AENOR, 2012) UNE 66181:2012 standard, which sets out a quality model based on a set of indicators that represent three satisfaction factors of the recipients of online education. Each of these factors break down into key quality dimensions, based on which actions can be taken in order to improve the corresponding factor: 1. Recognition of the training for employability. (This factor represents extent to which online education increases the student's ability to integrate into the labour market and improve their existing position in the said market.) 2. Learning methodology. (This factor defines the most appropriate conditions applicable to the learning process based on the defined objectives.) 3. Accessibility. (This factor seeks to determine to what extent online education is accessible, usable, and practicable with efficiency and effectiveness by anyone.) The Agency for the Quality of the University System of Catalonia (AQU, 2007), designed a guide for the internal self-assessment of online education, which is an assessment model for Higher Education online programmes. It is composed of five dimensions that do not differ much from the dimensions proposed for assessing traditional programmes: 1. Strategic position of the degree (internal and external); 2. Training programme (educational profile and its suitability); 3. Instructional design (teaching methodology, suitability of the activities, teaching organization, guidance system, and interpersonal communication systems); 4. Learning assessment (evaluative system); and 5. Results (academic, professional and personal dimension).
The quality of online programmes is also assessed by the National Agency for Quality Assessment and Accreditation of Spain (ANECA, 2012) at three different stages. The first stage consists of an initial assessment of the education programme. ANECA assesses the project of the programme through the accreditation programme VERIFICA. Once the programme is implemented, ANECA monitors its implementation and development by the university, using for this purpose a specific accreditation programme called MONITOR. The third stage consists of a periodical renewal of the accreditation of the education programme. The said renewal is carried out by ANECA by using an accreditation programme called ACREDITA. Table 2 shows the criteria used by ANECA to assess programmes at the three abovementioned stages. It should be noted that the said criteria are used by ANECA to assess both traditional and online educational programmes.

Methodology
The starting point for designing the model was a bibliographical/documental analysis of the components of e-learning education programmes and of guidelines regarding the assessment methods proposed for these programmes, as well as a specific bibliographical study of the standards, models and guides created in Spain and other countries in order to evaluate the quality of e-learning education (AENOR, 2012;ANECA, 2012;AQU, 2007;EFMD, 2006; European Foundation for Quality in eLearning [EFQUEL], 2012; Giorgetti, Romero & Vera, 2013;Marshall, 2006;OLC, 2002;and others).
Based on the results of the said analysis, a first draft of the model was created, composed of two variables, fourteen dimensions and ninety indicators. In order to facilitate the application of the indicators and determine its utility, a descriptive sheet was created for each of them which contained all of the components indicated by the standard UNE 66175 (see an example in Table 3). Taking into consideration that the model was the result of the author's own elaboration, it was necessary to validate it and determine its reliability through a methodological proceeding in order to determine the extent to which the model assessed the components it was supposed to assess.
According to Hernández, García, Padilla, García, and Leal (2012), validity refers to "the extent to which an instrument actually measures the variable that it intends to measure" (p. 201). Specialized literature proposes different types of validity and procedures to define the validity of a measurement instrument. All of these procedures apply formulas that allow to obtain reliable results and that guarantee the achievement of the set out goals (Pérez Juste, 1986). In our case, the process of validating the model was conducted through validation by experts and application of the model in the assessment of four online programmes.
Validation through expert judgement "consists, mainly, in asking several persons to cast judgement regarding an object, instrument, educational resource or to give their opinion regarding a particular element" (Cabero & Llorente Cejudo, 2013, p. 14). For this purpose, we asked 23 international experts from different countries (Spain, Portugal, Italy, Mexico, Chile, Paraguay, and Uruguay) to validate the model. The experts were selected based on their experiences in the field and on papers they authored. The experts were asked to determine the validity of each dimension and indicator of the model based on the assessment and evaluative judgment of its univocality, relevance, and importance. As for the indicators, the experts were asked to assess the suitability of the calculation formula and the relevance of the evidence.
An online form was designed in order to facilitate the validation of the model by the experts. The form was divided in three sections. In section 1 the experts were asked to provide certain personal data. Sections 2 and 3 were used to carry out the validation of the dimensions and indicators (see Figures 1,  2, and 3).   Each dimension and indicator contained a comment field that could be used by the experts to suggest modifications. This field could also be used to suggest improvements of the proposed model. After validating all the dimensions and indicators, the expert saved and sent the form using the buttons appearing at the bottom of the form.
We contacted the experts via email in order to ask for their cooperation in the validation process of the provisional model. We attached two documents to the email message: (1) a brief description of the 137 proposed model, its objective, and the operationalization of its variables; and (2) a document with instructions on how to fill in the online validation form. We asked the experts to respond via e-mail within one month.
The results of the validation provided by the experts through the online form were received in webpage format based on HTML5 technology, making its analysis more difficult. For this reason, the results were incorporated to a table generated by the Microsoft Excel program. Once the results were incorporated, the quantitative and qualitative validity of the model was verified.
The quantitative validity was verified using the Statistics programme in version 12 and included calculating the facial validity index, the contents validity index, and the interjudge reliability index for all the indicators composing the model. The qualitative validation of the model was verified by collecting all the comments made by the experts to justify their validation, as well as their suggestions for the improvement of the model.
Once the qualitative validation was completed, the results were triangulated with the results of the quantitative validation and specialized literature, which allowed us to make decisions regarding the maintenance, modification or removal of an indicator. For this reason, we created Tables with the specific findings related to the dimensions of the model and all the proposed indicators (see Table 4).

Experts:
Highly assessed indicator. Nevertheless, not all experts understand the accessibility of educational objectives.
According to Ardila (2011), the educational objectives of the programme should be duly described. According to ANECA (2015), the students enrolled in the online programme should be granted access to the relevant information regarding the programme and its educational objectives.

Researchers' decision:
1. The name of the indicator has been adjusted in order to facilitate its univocality. The new name is: The educational goals of the programme are available and accessible to all persons interested in the online programme.
The definitive model for the assessment of Higher Education online programmes was designed based on the results of the triangulation process. In order to verify the utility of the model, it was applied in the assessment of four online programmes.
The main purpose of this part of the study was to validate the model in an actual context and to verify its suitability for the self-assessment of online Higher Education programmes. To achieve this goal, four self-assessment protocols were designed (one for each of the assessed programmes) to be used by researchers as observation and assessment guidelines. The data was collected from the evidence described in the descriptive sheets for each indicator. By way of example, Table 5 shows a fragment of the protocol for one of the aforementioned self-assessments. The document "Planning of activities" posted in the virtual classroom of the programme. The document "Activities" posted for each unit of the programme.
All the activities are coherent with the educational goals. x The educational objectives of the assessed programmes have not been defined and, thus, it is not possible to verify whether the learning activities are coherent with the objectives.
The document "Programme syllabus." Interview with the coordinator for the degree in logistics and transport.
Instructions for each learning activity have been drafted.
x Each activity includes guidelines that describe it, as well as its purpose, the Web tools to be used and the assessment of the activity.
The documents entitled "Learning activities for Unit 1," "Learning Activities for Unit 2," and "Learning Activities for Unit 3." This application allowed us to verify the potential of the model while assessing the quality of the programmes through the detection of their strengths and weaknesses in order to design an action plan for their improvement.

An Integrative Model for the Assessment of the Quality of Online Higher Education Programmes
The main result of this research is the development of a model composed of two variables, fourteen dimensions, and eighty-one indicators.
Variables of the model. A variable is a characteristic or quality, magnitude or quantity, that can undergo changes and that is subject to analysis, measurement, assessment, or control during a research project (Arias, 2012). In the field of the assessment of the quality of education programmes, a variable is a characteristic that expresses the feature of the quality to be assessed. There are two variables considered in the integrative model for the assessment of the quality of online Higher Education programmes developed here. The first variable is the quality of the online programme itself. We consider this variable to be crucial, as it enables the assessment of the quality of all of the components of an online education programme. The second variable refers to the ongoing assessment of the online programme, which provides continuous information regarding its progress. This information enables the improvement the programme through feedback and self-adjustment.  Dimension 2-Online programme objectives: the proposals and objectives to be achieved by the programme.
 Dimension 3-Student profile: a set of the defined knowledge and competences that must be met by new students and those who have completed the programme.
 Dimension 4-Thematic contents of the e-learning programme: themes and topics that constitute the online programme.
 Dimension 5-Learning activities: different tasks carried out by students.
 Dimension 6-Online teacher profile: particular features that characterize the person who gives the online programme.
 Dimension 7-Educational material: any material used by the online teacher or by the student to facilitate the teaching-learning process.
 Dimension 8-Educational strategies: strategies used by the online teacher to support the teaching/learning processes.
 Dimension 9-Tutoring: a coaching process during the learning process that is carried out by the online teacher via individual attention.
 Dimension 10-Assessment of student's learning: the evaluation and monitoring of students.
 Dimension 11-Virtual platform: a software that allows educational contents to be distributed and to carry out the online educational programme.

140
 Dimension 12-Initial assessment of the programme: an assessment carried out one week before the planned start of the programme in order to establish the degree to which the programme is prepared to begin, and to know whether it can be launched.
 Dimension 13-Processual assessment of the programme: an assessment of the programme at the midpoint of its course, done in order to maximize its development, identify its weaknesses, to make decisions about how to eliminate them.
 Dimension 14-Final assessment of the programme: an assessment carried out immediately after the completion of the online programme to determine the degree to which the programme achieved its educational goals, and to measure the effects of the programme and student satisfaction.

Quality indicators for online programmes.
Indicators are a set of characteristics or features that constitute or allow for the description and assessment of certain dimensions of a variable. They can be presented as guiding questions or a checklist that make apparent the achieved degree of quality.
The system of 81 indicators developed to describe and assess the dimensions of our model can be used as a tool by Spanish universities to verify the extent of the application of the standards, parameters, and recommendations aimed at guaranteeing and assessing the quality of online education programmes. Table 6 describes the distribution of the indicators based on the dimensions. The appendix to this paper describes the indicators selected to assess each dimension. Table 6 Distribution of the Indicators Based on the Proposed Dimensions The purpose of the indicators of dimension 1 is to assess whether the online programme successfully explains and justifies itself so that students can see that the programme is useful, and can understand it in terms of their learning process and professional development.
The indicators of dimension 2 are proposed as ways of assessing whether the programme defines clear and measurable objectives and whether these objectives are coherent with the syllabus of the programme and are responsive to the demands of the labour market.
The purpose of the indicators of dimension 3 is to assess whether the programme clearly defines its access and graduation profiles, and whether these are made available to the students.
The indicators of dimension 4 assess the quality of the thematic content of the programme. They assess, among other things, whether the content is appropriate, taking into consideration the study load and whether the content is coherent with the educational objectives and the graduation profile of the programme, as well as whether the different topics and subjects are up to date, easily understandable, and arranged in a logical manner.
The indicators of dimension 5 allow for the assessment of whether the learning activities meet the following quality criteria: different typology; contribution to the achievement of the educational objectives of the programme; encouragement of teamwork; definition of guidelines, schedules, deadlines, and delivery methods for each activity.
The indicators of dimension 6 refer to the qualifications necessary to teach the programme. These qualifications include the pedagogical and technological competences required for the online teaching/learning process.
The indicators of dimension 7 serve to assess the quality of the teaching materials. Among other aspects. These indicators assess whether the teaching materials are appropriate, sufficient, up-to-date, motivating, and accessible by students.
The indicators of dimension 8 assess the quality of the teaching strategies applied by the online teacher, focusing on their typology and coherence with the educational objectives of the programme.
The purpose of the indicators of dimension 9 is to assess the quality of the individual and group tutoring sessions carried out by the online teacher. Moreover, they assess whether the teacher provides the students with appropriate feedback for each activity.
The indicators of dimension 10 assess the strategies applied for the continuous assessment of the learning process undertaken by the students. These indicators assess, among other things, whether the programme introduces clear assessment criteria and rules, and whether it applies them while taking into consideration the nature of each learning activity.
The indicators of dimension 11 assess the quality of the virtual classroom and whether it allows to manage all stages of the programme, such as the preparation of the programme content, the implementation of the programme, the assessment of the abilities acquired by the students, and the assessment of the learning process.
The indicators of dimensions 12, 13, and 14 serve to assess the three stages of the programme, that is, its initial, development, and final stages.
By generating a complete model for quality assessment that includes indicators for online education programmes, and allows for the assessment of the pedagogical and technological components of the programme and of its planning, development, and results, we intend to promote an improvement process for the continuous adequacy of the programmes based on their quality objectives.

Results of the Validation of the Model
Being aware of the limitations of this type of paper, below we present a summary of the results of the qualitative validation of the dimensions (Table 7) carried out by the experts who validated them in relation to their univocality (U), suitability (S), and importance (I). We also present some comments and suggestions regarding the indicators of the model.
In order to affirm that a dimension was sufficiently valued by the experts, the following criteria were established:  0% to 69% positive responses: dimension insufficiently valued  70% to 79% positive responses: dimension sufficiently valued  80% to 89% positive responses: dimension highly valued  90% to 100% positive responses: dimension obtained maximum value The type of validity sought was "positive" or "negative." Two scales of validation were used: the dichotomous validation (yes, no) and Likert validation (values from 1 to 4). In this former, a "yes" was considered positive; a "no" was considered negative. In the latter, a 3 or 4 was considered positive; a 1 or 2 was considered negative. Moreover, a high score was considered as a criterion to affirm that a dimension was sufficiently valued by the experts. To that end, a minimum score of 70% of positive responses was established. Qualitative feedback received from the experts, which was not mandatory, was also used for the reformulation of the dimensions. As can be seen in Table 5, all 14 dimensions are considered clear, which means that they are precisely presented and understandable by the experts. Moreover, all dimensions are considered as highly suitable, that is, they allow, to a high extent, for the assessment of the quality of the online programme. Furthermore, the experts consider that all dimensions are very relevant, which means that they are considered as crucial for the assessment of the quality of online programmes.
Regarding this point, we would like to note that the differences in the percentages are due to the validation by the experts of the three validation criteria. The experts did not always give the same score for each validation criterion. For example, when it comes to dimension 1, 92% of the experts considered that the dimension was clearly defined (univocality), while only 83% of them considered it suitable and important for the assessment of the quality of the online programme. The percentages show the percentage of positive responses provided for each criterion. Table 8 shows the qualitative feedback provided by the experts regarding the dimensions of our model.  An appropriate dimension. We can see that programmes and courses are created and offered solely because it is possible to do so and not because they respond to the students' needs.
 It is neither relevant nor suitable.

2.
Educational goals of the online programme  Very relevant for the assessment of the quality of the programme.
 It does not mention the competencies and maybe it should mention them.
3. Student profile  The access profile does not appear in many programmes.
 I recommend that the name of the dimension be modified to "student profile" [previously called "access profile"].

4.
Thematic contents of the online programme  The updating and structuring of the contents are important.
 It is necessary to point out the criteria for the selection of the thematic content in order to assess whether they are appropriate or not.

Learning
activities  I do not consider it relevant.
 It should be kept in mind that, in some cases, students must carry out learning activities outside of the virtual classroom.

6.
Online teacher's profile  I think it is necessary to divorce the development of the thematic contents from the teaching activity, as often teachers are assigned programmes with already prepared teaching materials.

Learning materials
 This dimension is very useful.
 If the materials are outdated they cannot be used in the learning process.
 The materials must be coherent with the educational objectives.
 It is mandatory.
 I think that this is the aspect with the most influence on the quality of the online programme. 14. Assessment of the final stage of the online programme  It should not be limited to quantitative indicators.
As for the indicators of each dimension, all of them were assessed by the experts as unambiguous in their semantic definitions and relevant with respect to the dimensions in which they were included. Moreover, almost all of them were considered important for the assessment of Higher Education online programmes. Only nine indicators were assessed as unimportant; these were removed from the model: 1. Selection criteria for the thematic contents of the programme (The experts consider that the contents should be selected based on the knowledge and experience of the programme designers.) 2. The suitability of the learning activities with respect to online education (According to the experts, students should not be deprived of the possibility of carrying out practical activities in on-site mode.) 3. Requirements regarding the knowledge and use of the Information and Communication Technologies (ICTs) by the candidates to the online programme (According to the experts, students complete a propaedeutic course during which they acquire knowledge of ICTs and the ICT skills sufficient to enroll in the online programme.) 4. Qualifications of the online teacher regarding online education (According to the experts, this indicator should not be a requirement, but is a desirable feature of the online teacher.) 5. Qualifications of the persons in charge of the development of the programme in relation to online education (According to the experts, this indicator should not be a requirement, but is a desirable feature of the online teacher.) 6. Suitability of the materials and didactic resources for disabled students (The experts considered that this is not a universal element of the quality of online programme given that there are different degrees of disability).
7. Variety of didactic materials (The experts consider that a wider variety is not a sign of quality.) 8. Variety of didactic resources (The experts consider that a wider variety is not a sign of quality.) programme objectives, student profile, thematic contents, learning activities, online teacher profile, didactic material, learning strategies, learning assessment, tutorial, online classroom) and the assessment of its planning, application, and final stages. Various indicators were proposed for each dimension as an example of what should be assessed. The application of these indicators depends on the data collected and managed by the university, as well as on the university's available means.
The model presented here is addressed to the persons in charge of implementing online programmes, and to programme directors/coordinators. It can also be applied by online programme designers as a guideline for creating high quality programmes. It can also be useful to those who supervise the quality of Higher Education, as it is a technical document that allows for its revision and the drafting of a final report.
The model also provides a framework for future research, which will need to identify the elements of the model are stable, and those that can be adjusted to the specific context of each university in different countries. And given the variability of the conditions of the quality of online education, there is always a need for research that aims at these kinds of dimension and indicator adjustments.

The System of Indicators for the Quality Assessment Model for Online Higher Education Programmes
Variable 1: The assessment of the quality of the education online programme itself

N° Dimension
Indicator 1 Online programme justification The online programme has been created in relation to the educational needs existing within the society. The online programme responds to the labour market. The reasons that justify the necessity of enrolling into the programme by the student are duly explained. 2 Educational goals of the online programme The educational goals of the programme are adequate to the demands of the labour market. The educational goals are drafted based on the skills to be acquired by the students after completing the programme. The educational goals are coherent with the contents of the programme. The educational goals of the programme are measurable. The educational goals of the programme are available and accessible to all persons interested in the programme. 3 Access and graduation profile

Access profile
The access profile has been designed. The access profile describes the previous education required in order to enrol in the online programme. The access profile is accessible to all persons interested in the online programme.

Graduation profile
The programme contains the graduation profile. The graduation profile has been defined in terms of the competences to be acquired by the students after completing the programme. The graduation profile has been updated according to the needs. Verification of whether the graduation profile of the programme is accessible to all persons interested in the programme. 4 Thematic contents of the online programme The thematic contents of the programme are appropriate taking into account the subject of the programme. The thematic contents of the programme are proportional to the study load indicated in the syllabus. There is coherence between the thematic contents of the programme and its educational goals. The themes and subthemes of the thematic contents of the programme are arranged in a logical manner. The thematic contents of the programme are reviewed on an annual basis. 5 Learning activities The learning activities proposed within the programme are of different types. All the activities are coherent with the educational goals. Instructions for each learning activity have been drafted. A schedule of the activities has been provided. A deadline for the delivery of each activity has been set out. The method and format of delivery has been described for each learning activity. The proposed activities promote collaborative learning. The volume of the learning activities is suitable to the teaching load of the programme. 6 Online teacher profile The online teacher has the appropriate professional profile according to the requirements of the programme subject.

153
The online teacher has the appropriate pedagogical competences in order to carry out the online teaching/learning process. The online teacher has the technological competences necessary to carry out the teaching/learning process. 7 Educational material The resources made available to the students are of different types. The programme offers the basic learning resources. The programme offers complementary learning resources. The programme contains a set of Web-based learning resources. All the learning resources are coherent with the educational goals of the programme. The learning resources of the programme have been selected based on clear selection criteria. The basic bibliography of the programme is described.
The complementary bibliography of the programme is described. The basic and complementary bibliography is updated (30% of the bibliographical suggestions is dated no longer than five years prior). 8 Educational strategies The online teacher uses different types of teaching strategies. The teaching strategies used by the online teacher are coherent with the educational goals. The online teacher promotes different learning strategies among the students. 9 Tutoring The functions of the online teacher and of the persons involved in the development of the programme are defined and accessible by students. The online programme includes instructions regarding the communication methods with the online teacher. The timetable of the tutoring sessions has been defined.
The online teacher carries out individual tutoring sessions.
The online teacher carries out group tutoring sessions. The tutoring sessions carried out by the online teacher are monitored. The students receive feedback regarding each task. 10 Assessment of students' learning Different strategies for the assessment of the students' learning process have been defined. The assessment criteria of the learning process are accessible by the students. The criteria to be used to grade the students' progress are detailed and accessible by the students. The students participate in the process of determining and assessing the achieved progress. 11 Quality of the virtual platform The virtual platform includes a tool to submit activities or files. The virtual platform offers tools that allow to manage the learning activities. The virtual platform includes tools that allow to create group tasks. The virtual platform contains tools that allow the students to create their own personal learning environments. The virtual platform offers tools for the asynchronous communication. The virtual platform contains tools for the synchronous communication. The students can view the results of the completed exercises and/or exams in the virtual platform. The virtual platform contains a tool that allows the online teacher to monitor and manage the students. The virtual platform contains a section describing the functionalities of all the tools available in itself. Variable 2: The ongoing assessment of the education online programme N° Dimension Indicator 12 Initial assessment of the online programme The programme is ready to be launched at least one week prior to its start. All the resources (human, financial and technical) are sufficient to guarantee the quality of the development of the programme. All the persons involved in the programme are ready for it to start. The virtual classroom is ready to be used one week prior to the programme being launched. The teaching strategies used by the online teacher are appropriate in order to achieve the educational goals of the programme. The learning activities are appropriate for the students to acquire the competences described in the graduation profile. The planning of the programme regarding the activities, the deadlines and the resources is met. The online teacher is involved in the execution of the programme. The teaching materials and resources are suitable to the students' expectations. Student's motivation is stimulated. Some of the programme goals have been achieved. 1 4 Final assessment of the online programme Level of achievement of the educational goals set out for the online programme.
Level of fulfilment of the planned learning activities. Level of students' satisfaction with the online programme. Level of online teachers' satisfaction. Impact rate of the online programme Performance rate of the online programme. Success rate of the online programme.