The Design and Psychometric Properties of a Peer Observation Tool for Use in LMS-Based Classrooms in Medical Sciences

Zohrehsadat Mirmoghtadaie; Mohsen Keshavarz; Mojgan Mohammadimehr; Davood  Rasouli

doi:10.19173/irrodl.v24i1.6689

Authors

Zohrehsadat Mirmoghtadaie Assistant Professor, Department of e-Learning, Virtual School of Medical Education and Management, Shahid Beheshti University of Medical Sciences (SBMU), Tehran, Iran
Mohsen Keshavarz Department of E-Learning in Medical Sciences, School of Paramedical Sciences, Torbat Heydariyeh University of Medical Sciences, Torbat Heydariyeh, Iran.
Mojgan Mohammadimehr
Davood Rasouli Assistant professor, Center for Educational Research in Medical Sciences (CERMS), Department of Medical Education, School of Medicine, Iran University of Medical Sciences, Tehran, Iran

DOI:

https://doi.org/10.19173/irrodl.v24i1.6689

Keywords:

blended learning, virtual education, psychometrics, validity, reliability

Abstract

In peer observation of teaching, an experienced colleague in the educational environment of a faculty member observes the educational performance of that faculty member and provides appropriate feedback. The use of peer review as an alternative source of evidence of teaching effectiveness is increasing. However, no research has been done in the field of tool design and development to peer review in classrooms that use a learning management system (LMS). This study used mixed methods. In the qualitative stage, after studying sources and interviewing professors active in virtual education, a question bank was prepared and a 26-item initial questionnaire created. In the quantitative stage, the psychometric properties of the developed instruments, such as the face, content, and structural validity, were examined, and reliability tests were performed. IBM SPSS Statistics (Version 20) was used for analysis. Five categories, including content preparation, content presentation, effective interactions, motivation management, and support services, and 26 subcategories were determined to be effective indicators in peer observation in LMS-based classes in medical sciences. During content analysis, 9 items were removed due to lack of necessary criteria. Then, using principal component analysis and varimax rotation in the present mode )Watkins, 2018), 5 components with eigenvalues higher than 1 were extracted, which explained a total of 70.55% of the total variance. The inter-cluster correlation coefficient (ICC) was 0.88. Thus, the peer observation measurement tool, designed with 17 expressions using the answer method “yes/no”, showed good validity and reliability. The research results demonstrate that the evaluation of virtual classes of professors by their peers is effective and that the results can be used in e-learning promotion plans.

Author Biographies

Zohrehsadat Mirmoghtadaie, Assistant Professor, Department of e-Learning, Virtual School of Medical Education and Management, Shahid Beheshti University of Medical Sciences (SBMU), Tehran, Iran

Dr. Zohrehsadat Mirmoghtadaie is an assistant professor of e-learning in medical education at Shahid Beheshti University of Medical Sciences. She gained her Ph.D. in medical education in 2016. Her academic interest areas are open and blended Learning, medical education futures, e-learning, and content production in medical education. She is the head of the e-learning department and has ample experience in content production.

Mohsen Keshavarz, Department of E-Learning in Medical Sciences, School of Paramedical Sciences, Torbat Heydariyeh University of Medical Sciences, Torbat Heydariyeh, Iran.

Dr. Mohsen Keshavarz has a Ph.D. in virtual education planning. He is a faculty member and the Director of the Virtual Education Center of Torbat Heydariyeh University of Medical Sciences now. His research interests include online and virtual learning, blended learning, telemedicine, new educational technologies, e-health, and multiple literacies in online environments. He has published several articles on virtual education in international Journals such as IRRODL. He is an energetic advocate of distance learning in his home country of Iran, having translated Tony Bates’s book Teaching in a Digital Age to Persian in addition to several other projects, some with international collaborators. He has recently been introduced by Leaders & Legends of Online Learning as an international figure in the field of online learning.

Mojgan Mohammadimehr

Department of Laboratory Sciences, Faculty of Paramedicine, AJA University of Medical Sciences, Tehran, Iran

Davood Rasouli, Assistant professor, Center for Educational Research in Medical Sciences (CERMS), Department of Medical Education, School of Medicine, Iran University of Medical Sciences, Tehran, Iran

Davood Rasouli is a faculty member of the Center for Educational Research in Medical Sciences (CERMS) at the Iran University of Medical Sciences. He has a Ph.D. degree in medical education. His research interest areas are teaching and learning in medical education.

References

Akkoyunlu, B., & Soylu, M. Y. (2006). A study on students’ views on blended learning environment. Turkish Online Journal of Distance Education, 7(3), 43–56. https://dergipark.org.tr/en/pub/tojde/issue/16925/176657

Al‐Qahtani, A. A. Y., & Higgins, S. E. (2013). Effects of traditional, blended, and e‐learning on students’ achievement in higher education. Journal of Computer Assisted Learning, 29(3), 220–234. http://dx.doi.org/10.1111/j.1365-2729.2012.00490.x

Amira, R., & Jelas, Z. M. (2010). Teaching and learning styles in higher education institutions: Do they match? Procedia-Social and Behavioral Sciences, 7, 680–684. https://doi.org/10.1016/j.sbspro.2010.10.092

Andone, I., & Sireteanu, N.- A. (2009). Strategies for technology-based learning in higher education. The FedUni Journal of Higher Education, 4(1), 31–42. https://ssrn.com/abstract=1337962

Bates, A. W. (2022). Teaching in a digital age: Guidelines for designing teaching and learning (3rd ed.). Tony Bates Associates Ltd. https://opentextbc.ca/teachinginadigitalage/

Benner, P. (2012). Educating nurses: A call for radical transformation— How far have we come? Journal of Nursing Education, 51(4), 183–184. https://doi.org/10.3928/01484834-20120402-01

Bernstein, D., Burnett, A. N., Goodburn, A. M., & Savory, P. (2006). Making teaching and learning visible: Course portfolios and the peer review of teaching (1st ed.). Jossey-Bass.

Butt, B. Z., & Ur Rehman, K. (2010). A study examining the students’ satisfaction in higher education. Procedia-Social and Behavioral Sciences, 2(2), 5446–5450. https://doi.org/10.1016/j.sbspro.2010.03.888

Clipa, O. (2011). The profile of the academic assessor. Procedia-Social and Behavioral Sciences, 12, 200–204. https://doi.org/10.1016/j.sbspro.2011.02.027

Collis, B., & Moonen, J. (2012). Flexible learning in a digital world: Experiences and expectations (2nd ed.). Routledge.

Cunningham, I., Johnson, I., & Lynch, C. (2017). Implementing peer review of teaching: A guide for dental educators. British Dental Journal, 222(7), 535–540. https://doi.org/10.1038/sj.bdj.2017.316

Dill, D. (2007). Quality assurance in higher education: Practices and issues. In P. P. Peterson, E. Baker, & B. McGaw (Eds.), International Encyclopedia of Education (3rd ed., pp. 377–383). Elsevier.

Donnelly, R. (2017). Blended problem-based learning in higher education: The intersection of social learning and technology. Psychosociological Issues in Human Resource Management, 5(2), 25–50. https://doi.org/10.22381/PIHRM5220172

Drisko, J. W., & Maschi, T. (2016). Content analysis: Pocket guides to social work research methods. Oxford University Press.

Fernandez, C. E., & Yu, J. (2007). Peer review of teaching. The Journal of Chiropractic Education, 21(2), 154–161. https://doi.org/10.7899/1042-5055-21.2.154

Fletcher, J. A. (2018). Peer observation of teaching: A practical tool in higher education. The Journal of Faculty Development, 32(1), 51–64. http://doi.10.13140/RG.2.2.19455.82084

Garrison, D. R. (2016). E-learning in the 21st century: A community of inquiry framework for research and practice (3rd ed.). Routledge.

Gehringer, E. F., Chinn, D. D., Pérez-Quiñones, M. A., & Ardis, M. A. (2005). Using peer review in teaching computing. ACM SIGCSE Bulletin, 37(1), 321–322. https://dl.acm.org/doi/10.1145/1047124.1047455

Henrie, C. R., Bodily, R., Manwaring, K. C., & Graham, C. R. (2015). Exploring intensive longitudinal measures of student engagement in blended learning. The International Review of Research in Open and Distributed Learning, 16(3), 131–155. https://doi.org/10.19173/irrodl.v16i3.2015

Ibrahimzadeh, I., Zandi, B., Alipour, A., Zare, H., & Yazdani, F. (2010). The kinds of e-learning and different forms of interaction on it. Interdisciplinary Journal of Virtual Learning in Medical Sciences, 1(1), 11–22. https://ijvlms.sums.ac.ir/article_46026.html

Johnston, A. L., Baik, C., & Chester, A. (2020). Peer review of teaching in Australian higher education: A systematic review. Higher Education Research & Development, 41(2), 390–404. https://doi.org/10.1080/07294360.2020.1845124

Kay, R. H., & Knaack, L. (2008). An examination of the impact of learning objects in secondary school. Journal of Computer Assisted Learning, 24(6), 447–461. https://doi.org/10.1111/j.1365-2729.2008.00278.x

Keig, L. (2000). Formative peer review of teaching: Attitudes of faculty at liberal arts colleges toward colleague assessment. Journal of Personnel Evaluation in Education, 14(1), 67–87. https://link.springer.com/article/10.1023/A:1008194230542#citeas

Keshavarz, M., & Ghoneim, A. (2021). Preparing educators to teach in a digital age. The International Review of Research in Open and Distributed Learning, 22(1), 221–242. https://doi.org/10.19173/irrodl.v22i1.4910

Keshavarz, M., Mirmoghtadaie, Z., and Nayyeri,S. (2022). Design and validation of the virtual classroom management questionnaire. The International Review of Research in Open and Distributed Learning, 23(2), 121–135. https://doi.org/10.19173/irrodl.v23i2.5774

Kintu, M. J., Zhu, C., & Kagambe, E. (2017). Blended learning effectiveness: The relationship between student characteristics, design features, and outcomes. International Journal of Educational Technology in Higher Education, 14(1), 1–20. https://doi. 10.1186/s41239-017-0043-4

Lacasse, Y., Godbout, C., & Sériès, F. (2002). Health-related quality of life in obstructive sleep apnoea. European Respiratory Journal, 19(3), 499–503. https://doi.org/10.1183/09031936.02.00216902

Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975. Tb01393.x

Lee, J.- K. (2009). The effects of self-regulated learning strategies and system satisfaction regarding learner’s performance in e-learning environment. Journal of Instructional Pedagogies, 1, 30–45. https://aabri.com/manuscripts/08053.pdf

Min, H.-T. (2006). The effects of trained peer review on EFL students’ revision types and writing quality. Journal of Second Language Writing, 15(2), 118–141. https://doi.org/10.1016/j.jslw.2006.01.003

Missildine, K., Fountain, R., Summers, L., & Gosselin, K. (2013). Flipping the classroom to improve student performance and satisfaction. Journal of Nursing Education, 52(10), 597–599. https://doi.org/10.3928/01484834-20130919-03

Mohammadbeigi, A., Mohammadsalehi, N., & Aligol, M. (2015). Validity and reliability of the instruments and types of measurements in health applied research. Journal of Rafsanjan University of Medical Sciences, 13(12), 1153–1170. http://journal.rums.ac.ir/article-1-2274-en.html

Neuendorf, K. (2017). The content analysis guidebook. SAGE. https://dx.doi.org/10.4135/9781071802878

OECD (2021) The state of school education: One year into the Covid pandemic Paris, France: OECD

Opre, D., Calbaza-Ormenisan, M., & Opre, A. (2011). University teaching: Didactic expertise reflected by metacognitive abilities and emotional control. Procedia-Social and Behavioral Sciences, 29, 670–677. https://doi.org/10.1016/j.sbspro.2011.11.291

Patton, M. Q. (1983). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches [Review of the book Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches by E. G. Guba & Y. S. Lincoln]. The Journal of Higher Education, 54(3), 339–342. https://doi.org/10.1080/00221546.1983.11778201

Pinto-Llorente, A. M., Sánchez-Gómez, M. C., García-Peñalvo, F. J., & Casillas-Martín, S. (2017). Students’ perceptions and attitudes towards asynchronous technological tools in blended-learning training to improve grammatical competence in English as a second language. Computers in Human Behavior, 72, 632–643. https://doi.org/10.1016/j.chb.2016.05.071

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.10.1002/nur.20199

Ruiz, J. G., Candler, C., & Teasdale, T. A. (2007). Peer reviewing e-learning: Opportunities, challenges, and solutions. Academic Medicine, 82(5), 503–507. https://doi.10.1097/acm.0b013e31803ead94

Snyder, M. M. (2009). Instructional-design theory to guide the creation of online learning communities for adults. TechTrends, 53(1), 45–57. https://doi.org/10.1007/s11528-009-0237-2

Speer, S. (2010). Peer evaluation and its blurred boundaries: Results from a meta-evaluation in initial vocational education and training. Evaluation, 16(4), 413–430. https://doi.org/10.1177/1356389010382265

Stevens, J. P. (2012). Applied multivariate statistics for the social sciences (5th ed.). Routledge.

Twomey, A. (2004). Web-based teaching in nursing: Lessons from the literature. Nurse Education Today, 24(6), 452–458. https://doi.org/10.1016/j.nedt.2004.04.010

Waltz C. F., & Bausell R. B. (1981). Nursing Research: Design statistics and computer analysis. F. A. Davis.

Watkins, M. W. (2018). Exploratory Factor Analysis: A Guide to Best Practice. Journal of Black Psychology, 44(3), 219–246. https://doi.org/10.1177/0095798418771807

Westen, D., & Rosenthal, R. (2003). Quantifying construct validity: two simple measures. Journal of Personality and Social Psychology, 84(3), 608–618. https://doi.org/10.1037/0022-3514.84.3.608

Wood, D. F. (2003). ABC of learning and teaching in medicine. Problem-based learning. BMJ, 326, 328–330. https://doi.org/10.1136/bmj.326.7384.328

Yong, A. G., & Pearce, P. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. The Quantitative Methods for Psychology, 9(2), 79–94. https://doi.org/10.20982/tqmp.09.2.p079