Multi-Country Experience in Delivering a Joint Course on Software Engineering – Numerical Results

A joint course, created as a result of a project under the auspices of the ‘Stability Pact of South-Eastern Europe’ and DAAD, has been conducted in several Balkan countries: in Novi Sad, Serbia, for the last six years in several different forms, in Skopje, FYR of Macedonia, for two years, for several types of students, and in Tirana, Albania, in the form of a crash, seven-day course, for the last two years. In this paper, we will put an emphasis on the assessment methods used within these courses, and compare them with the ‘original’ course that has been conducted at the Humboldt University in Berlin for almost a decade. Having a good environment for comparisons we draw some conclusions about teaching software engineering in different environments.


Introduction
Coinciding with the trends in European high education, under the auspices of the 'Stability Pact of South-Eastern Europe' and 'DAAD, Deutscher Akademischer Austausch Dienst' ('German Academic Exchange Service'), a project was established in 2001. The main idea of the project was to create and develop common courses in several fields of computer science. Also, the intention of the project was to enable the usage of shared materials for courses at a wide range of universities in countries participating in the project.
The project joined participants from 15 universities and nine countries: Germany, Serbia, FYR of Macedonia, and Bulgaria being the core members, and Croatia, Bosnia and Herzegovina, Romania, Albania, and Montenegro, as associate members. More about the project, its goals, and members can be found in the Joint SE course homepage (2013), as well as publications by Zdravkova, Bothe, and Budimac (2003) and Bothe et al. ( , 2005, while the experiences gained were described by Budimac et al. (2008Budimac et al. ( , 2009Budimac et al. ( , 2011.
The general goal of the project is improvement and adjustment of educational processes in South-Eastern Europe, with respect to the current and modern trends of countries within the European Union, and from the start it managed to fulfil several more specific goals: • inclusion of 'Software Engineering' as a stand-alone course and a core course in universities' curricula of participating countries; • consensus in creation of a joint course, including determination and selection of appropriate topics to be included as the basis for the common pool of topics; • creation and development of joint teaching, examination, and assessment material for the selected topics: slides, case studies, assignments, exam questions, literature...; • establishment of e-Learning facilities, used both as a simple repository of teaching materials and more subtle materials in a form of e-Lessons; • establishment of a research and education framework as the basis for future educational and scientific cooperation.
These goals are implemented through cooperation in creation, improvement, and enhancement of teaching materials, and production of a distributed, Internet-based, multilingual university course. The joint course in software engineering originated from the course that has been conducted at the Humboldt University in Berlin for several years. Its main objective was to present introductory notions and principles of the discipline, including a wide spectrum of sub-areas suggested by the ACM and IEEE societies (Computing Curricula, 2001) and others (Bran et al., 2001). Thus, the course Multi-Country Experience in Delivering a Joint Course on Software Engineering -Numerical Results Budimac, Putnik, Ivanović, Bothe, Zdravkova, and Jakimovski Vol 15 | No 1 Feb/14 86 covers more than 85% of the basic lessons suggested in 'Curricular guidelines for undergraduate programs in computing.' Since this paper will deal mostly with exercises and assessment, it is interesting to notice that the course is accompanied by a pool of case studies discussed during lectures and processed through assignments. From this pool lecturers/instructors are free to select the most suitable one(s). In addition, there is a pool of assignments referring both to the course contents and case studies, thus forming the base for specific exercises.
Together with the assignments, sample solutions, correction hints, and typical errors are collected and presented to students when appropriate. Through these additions, flexibility is added to the course.
After practical experiences in running (almost) the same course in four countries with five educational directions, we collected numerical data that could enable understanding of possible differences between conducted courses with respect to students and their environment. Thus, the goal of this paper is not only to describe our project but to present our conclusions based on collected numerical data. In our efforts to understand possible differences we used a field research technique (collecting existing records) and partly also the survey research method. Both approaches belong to qualitative research methodologies whose main aim is to understand and possibly predict the behaviour of the phenomenon under investigation.

Related Work
Our international educational joint project is not alone in its endeavour to improve teaching, and there are other similar projects that can be found in books and research papers. There are for example three other projects dealing with the same field of software engineering, such as those explained by Modesitt (2002), Doberkat et al. (2004), and Hilburn et al. (2003). Others we can mention are Ariadne (2013) and Merlot (2013) which are involved in some other fields of computer science, but still gather participants from different countries.
What we see as a major difference between our project and those mentioned is the methodology of course creation. While the mentioned courses created a relatively independent set of courses, from which participants can choose and adjust those they prefer and can include into their curricula, our aim was to create the course as a whole, including complete teaching materials, lectures, assignments, case studies, exam questions, and even lists of typical errors made by students. By this approach, our aim was to ease material reusability and enable use of these resources even to those lecturers who do not have software engineering as a primary field of interest.
Since as a consequence and the follow-up of our project, within a Tempus project (TEMPUS course homepage, 2013), a whole curriculum for master studies in the field of software engineering was created, two other projects that had the same approach as this  Caplinskas and Vasilecas (2003), the idea of the MOCURIS project was to develop the whole project, regarded as "... a system composed of courses, modules, labs, projects, and other components." Pretty much the same problem we encountered is mentioned in this paper, and that is the problem of "existing staff abilities." As the authors say "... no one separate university is able to implement such curriculum separately, because of the shortage of human and other resources." The same solution was implemented by both projects, and that is accumulation of resources of the partners, exchange of not only ideas and opinions, but also of teaching staff as necessary.
The second project that started with the need to exchange experiences and views, but concluded with the development of the whole master curriculum, is mentioned by Tibaut et al. (2013). They created an e-Learning environment for an international master level program in information technologies for the field of architecture. This environment "... integrates resources (units of study, learning management system, virtual classroom, teachers and students) from five European universities." What differentiates this project from ours is the fact that while we aimed to combine expertise and experiences of lecturers, project members, and create joint courses, the ITC-Euromaster project decided to join individually and separately created courses, accepted by the project consortium, and offer their use to the other project members via an e-Learning system that is the integral part of a project. This meant also inclusion of lecturers through a video conferencing system, opposite to our approach where lecturers are present in person, until individual project participants develop their own teaching staff.

Structure of the Joint Course
During the last several years, the same version of the course has been conducted by some participants of the project.
The Humboldt University in Berlin has the longest tradition in conducting the course, where it has been conducted for more than a decade.
At the University of Novi Sad, Faculty of Sciences, the course has been conducted in several different ways: • for undergraduate students of computer science, eight years, • for undergraduate students of the direction 'Professor of Geography and Informatics', also eight years, • two years for postgraduate students of computer science. After that, as a part of Tempus project CD-JEP-18035-2003 (TEMPUS course Even though all of the lecturers had the freedom to choose the methods of course conduction, the basic structure of a course is rather similar at all universities, as agreed within the project.
The course consists of 28 topics that cover most of the introductory notions from the software engineering field. The complete list of 28 topics is given below.

Part I: Introduction to Software Engineering
Topic 1: What is software engineering? gives insight into motivation, areas of interest, definitions, and history of the field.
Topic 2: Quality criteria for software products covers classification and definitions of software quality criteria, mentioning standards dealing with questions of quality.
Topic 3: Software process models -introduction explains the activities of software development, gives overview of models, and introduces details on selected process models.
Topic 4: Basic concepts and software development documents gives overview of concepts and documents created during the software development process.

Part II -Requirements Engineering (Analysis and Definition)
Topic 5: Results of the 'Analysis and definition' phase -Feasibility study, product model, and requirement document are discussed here.
Topic 6: Cost estimation -Costs, factors, and presentation of the 'function point' method are covered in the topic.
Topic 7: Basic concepts of the function-oriented view is devoted to function-tree and data-flow diagrams.
Topic 8: Basic concepts of data-oriented view -Data dictionary and entity relationship concepts are presented in the topic. Topic 9: Basic concepts of rule-oriented view is concerned with concepts of rules, decision tables, and trees.
Topic 10: Structured analysis introduces notions of context diagram, DFD hierarchy, mini-specification, and shows the process of structured analysis.
Topic 11: Basic concepts of state-oriented view -Petri-nets, state automata, and activity diagrams are covered within this topic.
Topic 12: Basic concepts of scenario-based view is dedicated to concepts of collaboration diagrams and sequence diagrams.
Topic 13: Object-oriented analysis covers the transition process from object-oriented analysis to design, explaining activities, class diagrams, use cases, UML, and shows the process of object-oriented analysis, using case studies.
Topic 14: Formal software specifications and program verification -Z, algebraic, and Hoare logic are presented.

Part III -Design
Topic 15: Overview of design activities presents notions of software architecture, specification of components, quality assurance, giving an overview of some software architectures.
Topic 16: Structured design is mostly dedicated to structure charts, illustrating the process of structured design.
Topic 17: Object-oriented design covers the transition process from object-oriented analysis to design, explaining characteristic activities in more detail.

Part IV -Implementation and Testing
Topic 18: Implementation -Principles, methods, guidelines for the implementation phase of software development are presented.
Topic 19: Systematic testing gives a classification of testing methods, and discusses review/audit, control-flow and data-flow oriented methods.
Topic 20: Functional testing is dedicated to the most important testing methods, including a presentation of testing tools.

Part V -Advanced Topics
Topic 21: Software metrics introduces several metric methods: cyclomatic complexity, Halstead, LOC, object-oriented metrics, and presents several CAME tools. Topic 23: Reverse engineering discusses notions of software repair, reengineering, and restructuring, including presentation of CARE tools.
Topic 24: Quality of software development process and its standardization covers standards such as ISO 9000, and capability assessment models.
Topic 25: Introduction to software ergonomics -The most important notions of graphical user interfaces, standards, and guidelines are presented.
Topic 26: User manuals -Dedicated to principles and guidelines for writing user manuals.
Topic 27: Project management -Planning, organization, people management, and control are the most important notions covered.
Topic 28: Configuration management -The topic is dedicated to motivation, explains activities, and discusses CVS.
The second essential component of the course is usage of complex case studies.
Currently, there are two case studies in use, 'Seminar organization' and 'XCTL software', while the development of several additional ones is in progress. Case studies are used after the presentation of theoretical topics, and the main idea is to show students the ways of application of theory and skills gained earlier, on a real software product.
• 'Seminar organization' represents a software system (Balzert, 1998) created for management of a company dealing with creation, organization, and presentation of external courses: contacts with clients and companies, communication with lecturers and participants, hotels and travel agencies, and so on. At the moment, this case study is used in 10 topics as an illustrative example.
• The second case study represents the software system 'XCTL', a real, existing system, used at the Department of Physics, Humboldt University, Berlin. This legacy system has been reengineered, refactored, and partially rewritten, in order for some additional features to be added. The case study is used in four topics as an illustration.
The third essential component of the course is team assignments, and in this paper we will mostly concentrate on these. There is a pool of assignments created, which can be used both for illustration of theory and for testing the acquired knowledge. An agreement within the project has been reached that the assignments will be given to teams of students. Depending on the number of students, from 5 to 20 teams were created, numbering from 3 to 5 students. Members of the team are self-chosen which enables easier organization of teamwork, which is also a useful practice elsewhere (e.g., Bielikova and Navrat, 2004). After presentation of an appropriate topic, an assignment is given with a deadline of 2-3 weeks for solving. Teams have to organize meetings, discuss the assignment, distribute obligations, perform work, and submit a written report. Points are then assigned to these solutions and all of the team members receive the same number of points.
A minimum number of points required for a student to qualify for the final exam is settled at 50% of the possible maximum number of points. How those points influence the final grade is not the same: In Germany and Albania students were required to achieve 50% of the points, which qualifies them for the final exam, but does not influence the final grade. In Serbia and in FYR of Macedonia, besides the limit of 50%, the number of points influences the grades.

The Assignments
For the first time, during the school year 2004-05 a complete, absolutely identical course, with the same case studies and the same assignments, was held in Germany (Berlin) and in Serbia (Novi Sad).
In the following years, the Faculty of Natural Sciences and Mathematics of the 'Ss Cyril and Methodious' University of FYR of Macedonia in Skopje adopted the same lecturing style for the software engineering course. Later on, the course was conducted in a slightly different style at the Polytechnic University of Tirana, but with the same general structure. It was conducted as a seven-day crash course, where most of the topics were presented to students, including case studies, while only four assignments were given to students.
A pool of nine assignments has been created. They are: • Assignment 1: Review of 'preliminary requirements specification' and 'requirements specification.' Case study 'Seminar organization' is used, and both requirements specifications are part of it. Students have to find misunderstandings, discrepancies, and errors and write a report with suggestions on how to solve the problems.
• Assignment 2: Application of a function-point method. Again, requirements specification for 'Seminar organization' is used and students have to estimate costs, expressed in human-power, for development of this software.
• Assignment 3: Review of a product model resulting after structured analysis.
Data-flow diagrams for 'Seminar organization' software are presented, including • Assignment 5: Definition of a formal specification for several given operations.
After being introduced to formal specification of several classic operations, students have to develop their solutions for some additional operations.
• Assignment 6: Review of a solution for the fourth assignment of a different team.
Teams were faced with another teams' solution of assignment 4. Analysis of other teams' solutions exposes the students to a different view on the same problem.
• Assignment 7: Measuring the quality of given software. The implementation of 'Seminar organization' system is used as a case study to be measured.
• Assignment 8: Specification of a regression test. Students are required to develop a regression test package for a given example program, using the given testing tool.
• Assignment 9: Creation of a classification tree. Practicing usage of classification tree method for function-oriented tests. Students are expected to specify a systematic test for a given business process, from the requirements specification for the 'Seminar organization' case study.
In practice, the following procedure is applied: Teams are given specific tasks and are expected to produce results before a given deadline. Each member of a team has to read, think about, and reflect on a task before the team meeting. During several team meetings, the team discusses and solves the task. Occasionally, one class is organized where the most interesting and provocative solution is presented by the members of the team submitting it.
For assignments, students are divided into teams, according to their choice. This approach has several advantages. The first is simplicity from the managerial point of view. Second is that the opportunity for a student to sign up for a team of their choice creates a tendency to base the choice on personal relationships. Thus, time needed for adjustments and adaptation of team members is shortened. Thirdly, efficiency of teams created this way tends to be rather high.

The Assignment Results
In this section, we will present results of the assignments gained at different universities. We will start with the two groups of students from Novi Sad, continue with the results achieved in Tirana, then in Skopje, and finish with the results of Berlin students.

Novi Sad Students
Average results and number of points students gained per assignment give us some interesting insights. Let us first present results for two groups of Novi Sad students, students of the 'Computer Science' study programme, and students of the 'Professor of Geography and Informatics' programme. Even though the number of students constantly grows, the percentage of gained points for assignments shows a more or less regular behaviour.
Results of assignments for the first year are significantly different than those for the following years. We think that the reasons are twofold: inexperienced lecturers during the first year and inexperienced assistants who checked the solutions for the first time.
Results of assignments for the last year (gray-coloured area in the table) are also sometimes radically different. Since the same thing also happened in Skopje, we believe that the reason for this is the fact that during this year, the first generation of students studying according to Bologna principles approached this course. While this alone can influence results, the more important thing in our opinion is the fact that a lot of not-sogood students from previous generations finally made the effort of passing to the final year of their studies, in order to avoid switching to a new curriculum. Namely, the curriculum created in accordance with Bologna principles introduced several new courses, discarding of some of the old courses, changing the way courses are assessed, which was highly undesirable for older students.
The worst results and the lowest number of points are usually gained for assignment number 2 (application of the function-point method). Even though it is quite straightforward -a conclusion that has also been made by students -it seems that the assignment has enough hidden difficulties such that it is a regular practice that not many teams reach the maximum number of points. Assignment number 6 (review of a solution of another team's assignment) has the highest average, but it represents simply the ability of team members to defend their own opinion, so it is not of a high expertise level.
The really best results and the highest number of points are usually gained for assignment number 7 (measuring of the quality of software), denoted in Table 1 by a rectangle. Again, there are two reasons, in our opinion. It is again a straightforward and relatively simple task, where most of the required answers are given by the installed software. The second important reason is that this is the last assignment, given at the end of the semester, when students are aware and experienced in how and what they need to do to solve their task.
The assignment that requires the highest level of 'creative' work, assignment number 4 (creation of use-case and class diagrams), denoted in the table by an eclipse, also has a rather low average. The main reason for this we consider to be the lack of experience with real-life work, no practical abilities and skills, and possession of scholarly knowledge only.
The average total points achieved by students are sufficient for them to approach the second part of the exam, and even more it is close to 80% of possible points, which we consider very good. Worth noting is also assignment number 5 (definition of formal specification), but only in comparison with the solutions of other groups, so we will discuss this topic in more detail later. Here, we would just note that again, results from last-year students (gray area) are far lower than the results of any of the previous generations.
At the end, we note that the average of the total points for the last generation is again far below most of the others, and far below average. As a reminder, that is the generation consisting of the first enrolment of students studying by the Bologna principles, plus remains of previous generations, finally making it to their last year of study. What lies behind this, in our opinion, are several things. For 'Bologna' students • there is a much larger group of 'elective courses' for this generation of students, which naturally invited students to skip some of the more difficult ones that are needed for assignment solving; • there exists a much higher percentage of exam passing within this generation, because of changed methods of knowledge assessment. This in turn allowed a greater number of students to pass to the next study years, but with somewhat less knowledge, and considerably lower grades, on average.

For 'old' students
• they have been struggling with their studies for years, having lower knowledge and grades, passing exams only after several attempts and/or years of trying; • by the time they reached the final year, a high percentage of them had to find a job to support themselves, and their studies suffered greatly because of that.

Novi Sad Students -Professors Programme
Let us now look into the results of students of the Novi Sad programme 'Professor of Geography and Informatics'. The number of students in this programme is relatively small for making definitive conclusions, so we will just compare these results with those from the previous table.
Results of assignments for the first year are not much different from the results from other years, as they are for the students of the computer science programme. A possible reason for this is that they are both low enough, so there cannot be too much difference.  Table 2).
Again, average total points achieved by students were sufficient for them to approach the second part of the exam.
With this group of students, the difference for the last generation of students is not that obvious. The reason for this, in our opinion, is the fact that they do not all belong to the same generation, since (each year) more than half of these students remain from some of the previous years. This, in turn, may also partly explain why results for all of the assignments for this group are weaker than for the computer science programme.

Tirana Students
The second country where the course has been conducted is Albania. At the Polytechnic University of Tirana, during the school year 2006-2007, a seven-day crash course for the selected students of the 2 nd year of master studies was conducted by a professor from Germany and assistant from Serbia, for the first time. In the following years, the same crash course was conducted.
Because of the shortage of time and physical constraints, it has been decided that this group of students will have to solve only four assignments, namely assignments 1 (review of requirements specifications), 2 (function-point method), 5 (definition of algebraic specification), and 7 (measuring of the quality of software). Particulars of tasks were also slightly different -the first assignment they had to solve was before the course started in order to get introduced to the requirements specification. Three other assignments had to be solved after the course was finished, and they were given two weeks per assignment. The results are presented in Table 3.  again as expected, students studying in non-mother tongue achieved less than maximal results. As a consequence, master students were not much better than graduate students in total numbers.

Skopje Students
The third country that has been engaged in conducting a common course was FYR of Macedonia. The Faculty of Natural Sciences and Mathematics in Skopje started with the conduction of this course in 2007-08 and has been conducting it since. The data we present here concern two groups of students: students of 3-year studies and students of 4-year studies (according to Bologna principles). The following year besides these two groups another group appeared: students of 4-year studies, studying not according to Bologna principles. Here are the results. During the first year in Skopje only the first four assignments were used. The decision of the lecturers was that 'formal specifications' have been tested enough within other courses, while they did not manage to lecture on the topic of 'software metrics,' so this assignment could not be used. Instead, they added one more assignment from the field of 'product models,' which we will not consider here. During the second year, having more experience with the course, lecturers decided that the course would have six assignments. The first, obvious thing that could be noticed is the fact that students of the 4-year studies achieved much better results than students of the 3-year studies (except for the 3 rd assignment -Review of a product model). This is due to the general observation in Skopje that the students of 4-year studies are more skillful and possess a better background in informatics than students of 3-year studies.
Here again we can notice a big difference between results for the first and the second year of conducting the course. For example, for the first year (Table 4, (Table 4, white area), differences are not that high, which again might have to do with the experience of the assistant. For this school year, of much greater importance is the difference between groups of students. Students of the 3-year and 4-year studies (studying according to Bologna principles) have lower results for each assignment than the 'old' students. In some cases, that difference is extreme -for assignment number 7, they received only 35% and 13% of points, while the 'old' students received 82% of points on average! The difference is significant also for assignments number 5 and 6, and considerable for assignment number 4. This, being rather similar to the situation in Novi Sad, keeps the question about peculiarities and lower results of Bologna students still open.

Berlin Students
How do all of the mentioned results compare to Berlin students? For the course conducted in Berlin, statistics for three years are available (Table 5).  Table 5 Assignment Points for Berlin Students Except for the first assignment, percentages for Berlin students are quite different between years. As can be seen in the gray area of Table 5 (Table 6). Table 6 Test

Points for Novi Sad Students of Computer Science
Since the 3 rd test during school year 2010-11 (Table 6, gray area), we can notice significantly lower results than before.
The explanation is twofold. Throughout the year, current standings for all of the students are known and published, together with the final grade gained up to that moment. More importantly, students are aware that after the semester, there is one additional possibility to do the test(s) again. Those who passed some tests, but are not satisfied with their success, must either accept these points, or re-take all of the tests. So as the last test approaches, students start calculating their possibilities: If they estimate they will not get as many points as they want, they submit an 'empty' test, fail it, and retake it later. The results of tests obtained by another group of students from Novi Sad, professors of Geography and Informatics, are as follows (Table 7).  Table 8.  Table 8 Test

Points for Tirana Master Students
The number of points gained is much lower than the number of points gained by Novi Sad students of computer science and only slightly higher than points gained by Novi Sad students of the 'Professor of Geography and Informatics' programme. The only reasonable explanation, which was also confirmed through consultations with master students, was the problem of the English language. Usage of non-mother tongue in tests and answers presented the greatest problem to the students, greater than the expert knowledge required. During tests English had to be used on-site, both for understanding the questions and for answering, which created a lot of problems. The additional problem was the fact that the test was performed on the distance. So, ambiguousness within questions, even lingual ones, could not be solved.
In Skopje, the final part of the exam was organized through three tests. As in Novi Sad, students that failed a certain part of the exam, or were unable to take it, had one additional chance to take it. The results of tests obtained by students from Skopje are given in Table 9.  Table 9 Test Points for Students from Skopje Similar to the assignments, students of the 4 th year gained significantly more points than students of the 3 rd year. The distribution of points between tests is quite regular, the first one being 'the most difficult one' which is the common practice, before students get acquainted with the material.
As far as the percentages are concerned, actual comparison is not easy, because of the  (Table 10). We already discussed reasons for that in the previous section.

Table 10
Final Test Percentages for all Countries If we just extract the data for comparable groups, the results are more similar, as we think they should be (Table 11).  In the end, let us present students' average final grades for all courses conducted (Table   12; the scale of the grades is from 6 to 10).

Table 12
Final Grades In our opinion, final grades represent steady and expected trends. It is not just the expected average, it is also within the normal (Gaussian) distribution for each group.
Grades are significantly higher for students of computer science than for the programme 'Professor of Geography and Informatics' and for the 3-year studies.

Conclusions
The general opinion of all project members is that the project was very successful and useful, first of all for students. During project realization we also went a step ahead and developed and intensively used several educational tools within several programming courses. Also, we conducted several questionnaires in order to obtain students' opinion about different educational issues and influences like gender influences on studying, usefulness of wikis for students' practical team work, privacy and usability aspects of using LMS, and so on. The main achievements and results of these activities are: • Time for course preparation is drastically shortened. • Students are enabled to learn in accordance with contemporary contents, principles (Klašnja Milićević et al., 2011;Vesin et al., 2013), and European standards.
• Course compatibility, both general and concrete, is achieved.
• An excellent base for usage of distance learning principles is created Ivanović et al., 2013a).
• Experiences, methods, and learning activities and styles of lecturers from several different countries are adopted .
• Possibilities for different kinds of future cooperation among the project participants are promoted and recognized. What is more, based on excellent experiences, and using the same technique, we developed several other courses and introduced them into curricula of a number of universities (Ivanović et al., 2013b ).
The central attention in this paper has been devoted to the software engineering course.
The same course was conducted in four different countries over multiple years and the same pool of assignments was used. Furthermore, the lecturer in Berlin and Tirana was the same, as well as the assistant in Novi Sad and Tirana. In all countries we had students of computer science and with significant background in informatics, while in two countries we had students with lower background in informatics ('professors' in Novi Sad and 3-year students in Skopje). All this created a good environment to draw some conclusions based on numerical results.
• To be successful (student) in software engineering, one must have a certain background in informatics. This justifies the position of this course at the end of studies, rather than near the beginning.
• Assistants' experience in conducting the course is an important factor -every time the assistant changed, the results showed inconsistency with general patterns and trends in results.
• Although students of 'Professor' and '3-year' studies are getting significantly fewer points than students of computer science, their results are still proportional and follow the general pattern. Furthermore, comparable groups of students from different countries have quite similar results. This leads us to the conclusion that problems and issues of software engineering are universal, and that cultural, historical, and other differences do not influence students' achievements nor should influence teaching methods.
We also have to note that students studying according to Bologna principles exhibited significantly lower achievements. While we tried to explain it in the paper, we are still not ready to draw formal conclusions. Further investigation of this phenomenon is needed.