Evaluation of the Vocational Education Orientation Programme (VEOP) at a University in South Africa

To address the training needs of Further Education and Training college (FETC) lecturers, and in the absence of a full professional education qualification, several higher education institutions, FETCs, and other bodies in South Africa formed an alliance to develop a short programme towards a possible future full qualification. In 2010 a Vocational Education Orientation Programme (VEOP) was piloted. In line with the responsibility for quality assurance, and the need to inform further developments in the training of FETC lecturers, the aim of this research was to evaluate the VEOP presented by the University of the Free State (UFS). To reach the stated aim, a two phase evaluative study was undertaken (1) to assess the individual modules, and (2) to holistically investigate the quality of the programme. Two questionnaires were used to gather data. The first set of data was collected at the completion of each of the six modules. For the second phase of the study, 48 lecturer-students were randomly selected more than a year after completion of the VEOP. The study identified a number of strengths and weaknesses of the VEOP. The results emphasise the need to carefully select tutors and train them to have an understanding of the FETC milieu, rethink the methodology employed in the education training of FETC lecturers, and redesign the modules’ contents to better reflect the FETC sector. The need to enhance student support and improve administration is also highlighted by the study. The results of the study may inform the development of a full qualification for FETC lecturers.


Article abstract
To address the training needs of Further Education and Training college (FETC) lecturers, and in the absence of a full professional education qualification, several higher education institutions, FETCs, and other bodies in South Africa formed an alliance to develop a short programme towards a possible future full qualification. In 2010 a Vocational Education Orientation Programme (VEOP) was piloted. In line with the responsibility for quality assurance, and the need to inform further developments in the training of FETC lecturers, the aim of this research was to evaluate the VEOP presented by the University of the Free State (UFS). To reach the stated aim, a two phase evaluative study was undertaken (1) to assess the individual modules, and (2) to holistically investigate the quality of the programme. Two questionnaires were used to gather data. The first set of data was collected at the completion of each of the six modules. For the second phase of the study, 48 lecturer-students were randomly selected more than a year after completion of the VEOP. The study identified a number of strengths and weaknesses of the VEOP. The results emphasise the need to carefully select tutors and train them to have an understanding of the FETC milieu, rethink the methodology employed in the education training of FETC lecturers, and redesign the modules' contents to better reflect the FETC sector. The need to enhance student support and improve administration is also highlighted by the study. The results of the study may inform the development of a full qualification for FETC lecturers. professional occupational categories and, if skilled, they will alleviate the massive unemployment problem in the country (DHET, 2012a;Powell, 2012). FETCs provide pre-service training to young students, up-skilling, and the retraining of adults, as well as supporting adult students through literacy and numeracy training (Powell, 2012). In 2013 more than 600,000 students were studying at 50 FETC in South Africa (Financial Mail, 2013). FETC lecturers are "a crucial component in the skills development challenge in South Africa" (Akoojee, 2008, p. 310). The work environments of FET lecturers are complex and continually changing (Williams, 2010). Increasing student numbers and broadening of programmes offered need to be matched by an improvement in the quality and quantity of the lecturers (Akoojee, 2008). The need for FET lecturers to develop contemporary skills in teaching, learning, and assessment (Williams, 2010) (DHET, 2012b). This is in line with international trends to move FETC lecturers' training towards professionalisation, standardisation, and certification (Papier, 2010a).
While the finalisation of a framework for FETC lecturer qualifications is still in process, the need for training these lecturers is a pressing concern. Papier (2010b, pp.  Cosser, Kraak, and Winnaar (2011) found that in 2010, 57% of FETC lecturers have less than a degree/higher diploma.
To address the training needs of FETC lecturers and in the absence of a full qualification, several higher education institutions, FETCs, and other bodies formed an alliance to develop a 30-credit short programme towards a possible future 120-credit Vocational Education Certificate (Level 5) (DHET, 2012b). In 2010, a Vocational Education Orientation Programme (VEOP) was piloted and continues to be delivered.
The VEOP aims to • initiate FET lecturers to the FETC policies, context, and skills environment; • enable FET lecturers to interpret curricula and plan relevant learning and assessment activities; • facilitate the discovery of appropriate integrated teaching and assessment strategies; • develop an understanding by FET lecturers of learners, learning processes, and a culture conducive to learning; and • enable FET lecturers to manage administration processes. (DHET, 2012b) Several principles underpin the VEOP, namely practical classroom application; relevant theory; respect for adult learners and their previous experience and contribution; cultural diversity, tolerance, language appreciation and inclusivity; innovative teaching and learning strategies; the context of a lecturer's subject specialisation; an understanding of the future workplace of learners; and assessment which should include classroom observation and guidance (DHET, 2012b).
The VEOP offered by the School of Open Learning (SOL), in conjunction with the Faculty of Education of the University of the Free State (UFS), is aligned with the abovementioned general aims and principles agreed upon by the aforesaid alliance. It is a 30-credit short course at the HEQF Level 5. The VEOP consists of 300 notional hours that includes contact time, self-study, classroom practice, and assessment. It offers 6 modules developed by the UFS, and a seventh developed by the Swiss-South African Cooperation Initiative. Table 1 provides the layout of the modules.  (Perraton, 2007), Turkey (Gültekin, 2009), India (Mishra, Vijayshri, & Garg, 2009), the United Kingdom (Sampong, 2009;Wilson, 2008), and Australia (Sampong, 2009).
Distance education as a complementary mode of delivery was initiated as an effort to overcome the challenges of access, equity, cost effectiveness, and quality for higher education (cf. Sampson, 2003). Perraton (2007) defines distance education as "an educational process in which a significant proportion of the teaching is conducted by someone removed in space and/or time from the learner" (p. 12). Sikwibele and Mungoo  Sampson (2003) found that distance education serves those who cannot or do not want to make use of traditional classroom teaching. Demanding professional commitments and family responsibilities often make attending a conventional, full-time, face-to-face course with fixed timetables impossible (Sampson, 2003). Sikwibele and Mungoo (2009) additionally list the following benefits of distance education: It allows the training of more people, it can be delivered to home and work sites, and it is learnercentred and affords students more control of the pace and learning styles. Distance education is thus convenient, flexible, and adaptable. This suits full-time professionals such as FETC lecturers. Distance education methods have been used to teach, develop, and support teachers for many years (Aluko, 2009;Mishra et al., 2009). The effectiveness of distance education for teacher training is well documented (cf. Aluko, 2009 The two most important problems associated with distance education are high attrition rates and quality assurance (Sikwibele & Mungoo, 2009). It is therefore understandable that quality assurance is an important research area in distance education (Zawacki-Richter, 2009). Programme evaluation is an essential aspect of quality assurance.

Evaluation of the Programme
The need for higher education institutions to evaluate themselves continuously is Sampong (2009), moreover, believes that successful programme development cannot occur without evaluation. In distance education, programme evaluation encompasses the evaluation of programme objectives, content, instructional design, support services, assessment practices, student achievement, and the use/impact of technology to improve the quality of teaching-learning, to enhance the relevance of the programme, to access how the programme is perceived by its stakeholders and meet their expectations, and to provide regular feedback on the factors that affect outcomes (Mishra et al., 2009). In the evaluation of distance education programmes, cognisance should also be taken of the American Distance Education Consortium's (ADEC) principles for good practice: • Design for active and effective learning by considering the needs and characteristics of the learners, the nature of the content, appropriate instructional strategies and technologies, and the desired learning outcomes.
• Support the needs of the learners by providing advisory, technical, and library support.
• Develop and maintain technological and human infrastructure.
Vol 14 | No 4 Oct/13 73 • Sustain administrative and organisational commitment to quality by integrating distance education into the mission, providing financial commitment, including faculty development and rewards, training to support those involved, and including marketing and management structures to promote and sustain distance education programmes. (Buford, 2005) Feedback instruments designed to help Australian training organisations to measure the quality of vocational education and training include items on trainer quality, satisfaction, effective assessment, clear expectations, learning stimulation, training relevance, competency development, training resources, effective support, and active learning (Coates, 2009).
VEOP presented by the SOL is essentially a distance programme, with limited contact sessions intended for FETC lecturers. The evaluation of the VEOP was consequently guided by ADEC's principles for good practice (Buford, 2005), as well as Sampong's (2009) and Coates's (2009) guidelines for distance and FET evaluation, respectively.

Aim of the Study
In line with the responsibility for quality assurance, and the need to inform further developments in the training of FETC lecturers, the aim of this research was to evaluate the VEOP presented by the UFS. To reach the stated aim, a two phase evaluative study was undertaken (1) to assess the individual modules, and (2) to holistically investigate the quality of the programme. The programme evaluation is aimed at improving the current programme, as well as informing the role-players towards the development of a full qualification. The DHET (2012b) anticipates that the VEOP will form an integral part of the Diploma in Vocational Education.

Research Method
Survey instruments have been identified as being the most popular data collection tools in outcomes measurements and evaluation (Aluko, 2009;Sampson, 2003). Two questionnaires were used to gather data about the student-lecturers' views of the VEOP at the UFS. The first questionnaire was developed by the staff of the quality assurance office of SOL. It has been used numerous times since 2005 to elicit the views of students on the different modules that form part of a wide variety of education programmes administered by SOL. The survey instrument used during the second phase of the study, compiled by the authors, was based on the content of the programme documentation submitted for accreditation. It was piloted in 2012 using 10 students from Bloemfontein (where the university is located) who did not form part of the sample in the study. The Vol 14 | No 4 Oct/13 74 pilot process enabled the researchers to improve some questions which appeared to be ambiguous.
Items in both questionnaires relied heavily on existing programme evaluation questionnaires and observed evidence (cf. Coates, 2009;Mishra et al., 2009;Sampong, 2009;Sampson, 2003;Taylor-Powell & Renner, 2000). The questions in the second survey were also based on the learning outcomes of the programme, as well as the principles that guide the programme (cf. Aluko, 2009). The researchers, furthermore, entered into lengthy discussions with the dean and the director of short courses at SOL to determine their perspectives on VEOP, their expectations of the evaluative study, and their views on the future of VEOP at SOL. They also acted as critical readers in the development of the questionnaire used during the second phase of the study. Insights gained from the discussions were incorporated into the questionnaire.
The data were captured and checked for accuracy by an experienced typist. The data were analysed, using the STATA IC11 software.
The data were analysed using the mean score, as a measure of centrality, and the standard deviation, as a measure of spread, to summarise the data. We then explored the differences in measurements between groups, using techniques that are typically used in inferential studies (the student's t-test and the one-way ANOVA). However, in this study, we do not claim inferences beyond this case; we used the techniques to merely differentiate between groups to gain partial insights into the VEOP programme at this institution.
The internal reliability of the questionnaires was checked by calculating the Cronbach's alpha coefficient. The overall Cronbach's alpha coefficient for the rated responses during the first and second phases of the study was calculated to be 0.9002 and 0.9532 respectively, which indicates that the responses have a high level of reliability.

Sampling and Data Collection
The first set of data was collected at the completion of each of the modules during 2011.
Each lecturer-student received a module evaluation form on completion of each of the first six modules, on which they could provide feedback on their experiences during the module. Of the 1,536 evaluation forms distributed to lecturer-students, 418 were returned of which 412 could be used.
The second set of data was collected from the same cohort of students during 2012 more than a year after the lecturer-students had completed the short programme. This survey was purposefully conducted after a substantial time had elapsed after completing the programme, to allow the lecturer-students to reflect on their experiences in the year following the programme. This survey focused on the quality of the programme, and not on specific modules, although the outcomes of the various modules were included as part of the purpose of the programme. For the second phase of the study, 60 lecturer-Vol 14 | No 4 Oct/13 75 students were randomly selected from three campuses, also randomly selected. They were telephonically invited to participate in the survey. Of the lecturer-students, 48 agreed to take part in the survey. The survey questionnaire was hand-delivered to each of them by an independent courier, commissioned by SOL, who had no connection or interest in the programme. The participants responded to the questionnaire at their convenience, sealed the questionnaire in the envelope provided, after which it was collected by the courier.

Ethical Considerations
Permission for this study was obtained from the dean of the SOL, as well as from the director of this programme. The participants' dignity, privacy, and interests were respected at all times. With the exception of a question asking the participants to indicate the campus at which they study/studied, the questionnaires did not contain any identifying aspects, names, addresses, or code symbols. Before completing the questionnaires, the participants were informed that the process was completely voluntary and that they could withdraw at any stage during the process. They were also informed during the first phase of the study that whether they chose to participate or not would have no effect on their grades or evaluation. The questionnaires were collected by administrators from the quality assurance office and not the tutors of the modules. They received no financial or other reward for taking part in this study.

Results of the Study
The most important results of the study are reported below, based on feedback from both surveys.
First Phase of the Study The first survey focused on the quality of the modules relating to four aspects, namely the content of the modules, the teaching and learning related to the module, the teaching material, and the assessment during the module. The lecturer-students had to relate their experiences of each of the modules on a 5-point Likert-scale (ranging from strong disagreement to strong agreement). As a score of 3 indicates neither a positive nor a negative response, and only 4 and 5 indicate an agreement, any mean score of less than 4 should raise concerns, in terms of the quality of the specific module.
In discussing the results, the four aspects related to the modules will first be analysed (Table 2); thereafter, a comparison will be made of how the various modules were rated (Table 3). Aggregated ratings are provided in Table 2.  From the combined score it can be seen that the respondents perceived the assessment to be the best aspect of the modules (M = 4.0867), followed by the content (M = 4.0336). The study material seems to need the most attention (M = 3.5231), and then also the teaching and learning aspects (M = 3.8808). The aggregated scores of the respondents on details regarding each of the four aspects are summarised in Table 3 (ranked from highest to lowest per aspect).  Respondents' ratings on various aspects of the different modules were also investigated.
The details are provided in Table 4.   The statistical significance of the differences in mean scores was explored, using the one-way ANOVA. No statistically significant differences were found between the study materials' rating in the different modules. Statistically significant differences were found between the content (p = 0.0000), teaching and learning (p = 0.0004), and assessment (p = 0.0002) ratings in the different modules. The Sheffé post-hoc test showed a number of statistical differences: • The content of module 1 scored significantly lower than that of module 3 (p = 0.000), module 4 (p = 0.025), module 5 (p = 0.008), and module 6 (p = 0.036). • The content of module 2 scored significantly lower than that of module 3 ( p= 0.049).
• The teaching and learning aspects of module 1 scored statistically significantly lower than those of module 3 (p = 0.037) and module 6 (p = 0.031).
• The assessment during module 1 was rated significantly lower than that of module 5 (p = 0.006).
• The assessment during module 2 was rated significantly lower than that of module 5 (p = 0.048).
While none of the modules scored below the point of neutrality (i.e., 3) on any of the aspects, in terms of quality assurance, SOL's aim is, at least, to be good (a score of 4).
While experiences pertaining to teaching and learning seem to be similar during the different modules, the content and assessment of module 1 and module 2 seems to need improvement, while these aspects of module 5 can be commended.
Whereas the evaluation during the first phase of the study focused on the various modules offered in the programme, and feedback obtained immediately after completion of the modules, the second phase of the study focused on the programme per se.

Results from the Second Phase of the Study
The questionnaire used during the second phase of the study contained mainly closed questions, in which the participants were requested to respond to statements on a 5point Likert-scale (ranging from strong disagreement to strong agreement). As a score of 3 indicates neither a positive nor a negative response, and only 4 and 5 indicate an agreement, any mean score of less than 4 should raise concerns, in terms of the quality of the specific programme.
The survey was aimed at four specific categories of issues pertaining to the VEOP, namely 1) issues generic to all programmes (standards, administration, mode of presentation, etc.); 2) issues pertaining to the campus where the lecturer-students attend the programme (administration, presentation, etc.); 3) the purpose of the programme; and 4) the principles underpinning the programme. Furthermore, the respondents' general perceptions of the module that they experience the most positively, compared to the module that they experience most negatively, were obtained.
The combined score on all the items in the questionnaire is 3.975 (with a standard deviation of 0.777), just below the benchmark of good (i.e., 4.000). The participants' Vol 14 | No 4 Oct/13 80 scoring on the four constructs are summarised in Table 5 (ranked from highest to lowest). From the responses, summarised in Table 5, it seems as if the principles that were stated indeed informed the programme (M = 4.0324), while the respondents were the least satisfied with generic issues pertaining to the programme (M = 3.9021). The details of these issues, on which the respondents were probed, are provided in Table 6. What was rated the highest by the respondents is that they felt accepted and included be noted that while all of the aspects explored scored above the point of neutrality (i.e., 3), many scored below the standard of good (i.e., 4).
To complement the module evaluations done during the first phase of the study, the modules were evaluated holistically: The student-lecturers were asked to rate the modules in terms of the module they rated the highest and the lowest. The results are displayed in two graphs.   While different lecturer-students rated the modules differently, the one module that stands out as problematic is module 1. It was rated the lowest by the majority of the participants, and scored the highest by only two participants. Using the rank order of Figure 1 and inverting the rank order of Figure 2, and then combining the two rankings, the rating of the modules is as follows: 1. Conducting and managing assessment ( While the students have different perspectives on which of the modules are the best, it is clear that module 1 is regarded as the module that needs the most improvement.

Discussion
The need for self appraisal in any programme is essential as it allows the institute offering the programme to reflect on the strengths and weaknesses of the programme.
The insights gained in the process allow for improvement, growth, and development.
This is even more important for any new programme and specifically a programme such as VEOP, where the training of FETC lecturers is not only a new field in Higher Education in South Africa, but still to be developed into a new complete qualification.
During the two-phase programme evaluation, a number of strengths emerged: During the evaluation of the different modules, the respondents were, in general, positive about the content of the various modules and indicated that they enhanced their knowledge.
The respondents were furthermore satisfied with the assessment during the modules and specifically indicated that the assessment tasks were marked and returned within a reasonable time. From the responses of the participants in the programme evaluation survey, it seems as if the principles that were stated were experienced by the lecturerstudents. The lecturer-students seem to appreciate the encouragement of the tutors towards student participation. What was rated the highest by the respondents is that they felt accepted and included during the course, that they experienced the VEOP relevant to their work, and that they perceived the study material to be of a good standard. These positives resonate well with ADEC's principles for good practice in distance education (cf. Buford, 2005).
However, not all experiences by lecturer-students were positive. A number of deficits in the programme became clear in this study. Lecturer-students indicated that tutors do not use teaching aids effectively. The additional learning material was indicated to be Vol 14 | No 4 Oct/13 84 insufficient, while the organisation of the module content needed some improvement.
Module 1 (the FET college: policy and workplace contexts), as well as module 2 (Methodology: integrated teaching and assessment) were rated the lowest during the module evaluation. Some issues were specifically pointed out as needing improvement, namely the content of modules 1 and 2, as well as the assessment, teaching, and learning during module 1. In the final programme evaluation, module 1 was again indicated as the one module with which the students are not satisfied. Respondents indicated that time to achieve the different modules' outcomes was not enough. Issues that scored the lowest by the respondents in the second survey were the administration of the VEOP at the UFS, and the acknowledgement and appreciation of previous experience.
Respondents who believe that they do not have enough time to achieve the module outcomes (cf. Printed self-instructional materials and the contact-sessions form the backbone of instruction in the VEOP. Therefore, evaluating various aspects of the module content, teaching material, and teaching and learning were concerns (Tables 3 and 4). This study reveals that although the students were, as a whole, satisfied with the module content of the different modules (Table 3), they grappled with the content and study material of module 1 (Table 4). The importance of unambiguous and sufficient learning materials should therefore be highlighted (cf. Mishra et al., 2009;Sampson, 2003). Mishra et al. (2009) furthermore stress that learning material, especially if it is lengthy and/or dense, "do(es) not fully substitute for the teacher" (p. 115). Gültekin (2009) notes that it is important that study material for distance learning is designed in accordance with distance teaching techniques that facilitate learning and are easily comprehended.
Student satisfaction with study material is not only a condition for successful distance education, but is fairly common in distance education. This study (Table 3), as well as programme evaluation studies by Mishra et al. (2009) and Gültekin (2009), found that students were satisfied with the design and content of the study material. On-site facilitators are critical to the effectiveness of a distance education programme (Sampson, 2003). It is therefore heartening that lecturer-students who took part in this study are relatively satisfied with the conduct of the tutors towards them, because the tutors encouraged them to participate in class, returned marked work within a reasonable time frame, and are positive about the tutors (cf . Tables 3 and 6). Sampson (2003) finds that support is a key issue in distance education. Students who took part in Sampson's (2003) study on the needs of M.Ed. education students were almost unanimous in stating their level of dissatisfaction with the amount of support they received. The lecture-students who took part in this study were also critical of the Vol 14 | No 4 Oct/13 85 administrative support they received. They evaluated the administration of the programme at the UFS, as well as on their respective campuses, as below the standard of good (i.e., 4) (cf. Table 6). Sampson (2003) believes that better co-ordinated administrative procedures that will allow for more effective communication between students and the university can reduce distance education students' frustrations. Sampson (2003) cites Rumble (1992) who states that the key to successful management of distance education lies in planning, organisation, leadership, and control.

Conclusions and Recommendations
Module evaluation as a standard procedure by the SOL is a strong point as it enables constant evaluation towards addressing the shortfalls in modules. Creating opportunities for lecturer-students to give feedback on the programme in retrospect is considered a strength, as it allows the participants to reflect on how the programme has impacted on their performance at the FETC where they teach.
The value of a study such as this is that it may lead to programme improvement. It is clear that the SOL has to rethink the methodology employed in VEOP, and redesign modules' contents to better reflect the FETC sector, in order to appropriately train FETC lecturers. It is clear that the selection of tutors is crucial and that tutors should have a thorough understanding of the FETC milieu, as well as pedagogies suitable for FETC lecturers. It is furthermore recommended that the amount of work that is expected during the VEOP needs to be revised. Too much work was put into the programme that is, in reality, a short programme. As the VEOP will become part of the first year of a full new education qualification, these modules should just introduce students to these topics, and allow for expansion and progression in the modules that will be built on them. Accordingly, the amount of worked crammed into the course and the assignments given should be reconsidered. Most importantly, it is recommended that all modules have to be reviewed and partially or completely rewritten, informed by the insights gained through this study. VEOP is not a qualification, but a short course designed to be part of the new education qualification for FETC lecturers. It does not focus on vocational improvement, but, rather, on promoting teaching practices, as well as classroom and learner management of lecturers who train students for vocations.
However, how it achieves the need to be enhanced in order to capture the nature and essence of the FETC sector more appropriately should be investigated. This study also highlighted the need to enhance student support and improve administration.
Cognisance should therefore be taken of Sampson's (2003) guidelines on how to improve planning, organisation, leadership, and control in distance education.
The important role FET can play in economic and individual prosperity is acknowledged worldwide (Coates, 2009 is still debating the development of a full qualification for FET lecturers. The study has revealed the strengths and weaknesses of the VEOP presented by the SOL, and showed that distance education is the only viable delivery mode for this short programme in a country with limited resources. Sub-Saharan Africa and other developing countries should thus consider using distance education as a workable option to enhance the competencies of their FET lecturers. Without a competent labour force developing countries will never be able to effectively compete in the global marketplace. The following limitations of this study should be noted: The sample for this study was a single (the first) cohort of students registered for one VEOP programme. Results from this study can therefore not be generalised. The low response rate during the first phase of the study (27.2%) and the small sample size during the second phase of the study (only 48 of the first cohort of 256 lecturer-students [18.8%]) may also inhibit the generalisation of results. Key principles for the evaluation of distance education, such as the use of technology, as well as technical and library support for students, were not investigated (cf. Buford, 2005). We furthermore used two self-constructed questionnaires, which does not allow us to claim consistency of measurement over time.
Despite these limitations, this study may inform the development of a full qualification for FETC lecturers. The FETC sector brings uniqueness to the education forum in the sense that it accommodates adolescent, youth, and adult learners. It also offers a rich mix of academic and vocation training that differs vastly from schools and universities.
Any education and training effort in this field should take this uniqueness into consideration in its design, development, and offering, even more so when courses are designed and offered to the FETC lecturer cadre.