April - 2002

Online Learning Activities: Beginning an international collaboration

Tom Nickel
Utah State University


Early Sunday morning, April 1, 2001, a U.S. Navy EP-3E Aries II surveillance plane collided with a Chinese fighter jet, causing the death of the Chinese pilot and forcing an emergency landing by the U.S. aircraft on Hainan Island, in the People’s Republic of China. Political relations between the United States and China had been deteriorating for months during the Spring of 2001 –- it was the stated intention of newly-elected President George W. Bush to move American policy toward China from “strategic partnership” to “strategic competition.”

Against this backdrop, Chinese and American students from three different graduate-level programs participated in a series of collaborative learning activities through the Internet, under the direction of the Department of Instructional Technology, at Utah State University (USU). Two of the programs were based in Utah: Twelve USU graduate students enrolled in an on-campus course in Distance Education, and a cohort of ten elementary and secondary in-service teachers in an online Distance Learning Endorsement (DLE) Program. The third group consisted of four English-speaking Chinese students in a Masters program in Educational Communications at South China Normal University (SCNU), Guangzhou, People’s Republic of China, as well as a recent graduate of that program.

Small as it was, it was, of course, the SCNU group that made the experience unique and highly motivating to the American participants. The online learning experience was an initial step in creating a broader and deeper joint program between USU and SCNU in the area of instructional technology. There were numerous obstacles, only some of which were expected. The political event that took place could not have been anticipated, and the incident seemed to lend even greater importance to the development of Web-based learning communities among people from different nations.

Background and Goals

The seeds of the USU-SCNU collaboration were planted at an educational conference on “Learning and Teaching Online” held in January, 2001 in Guanghzou, China, for which SCNU was the sponsoring university. At that conference, an Instructor in Distance Education at USU met with Wang Guo Jian, Vice President of SCNU and Xu Fuyin, Dean of the Faculty of Educational Communication and Technology, and proposed a small-scale cooperative project utilizing simple technology, such as email, at the beginning.

Both of the SCNU representatives were receptive to the idea, which involved no new curriculum development or university credits to negotiate, and no complex or expensive hardware. For Professor Xu, it was simply a matter of recruiting a group of students confident enough in their English language capabilities to feel comfortable participating. After the conference, the USU Instructor was already scheduled to teach two Distance Education classes back in the United States, one online and one classroom-style. Along with the Chinese students, these two groups would become the basis for the project.

Details were discussed during the month of February and the program began in early March, only six weeks after the initial meeting. As stated in a Memo of Understanding between the two universities, finalized on February 23, the purpose of the collaboration was:

To provide an opportunity for instructional technology students from USU and SCNU to participate in shared online learning activities. By working together in an online learning environment, students will find what their national cultures have in common and will also benefit from exposure to different points of view. It is expected that the USU-SCNU Distance Learning Collaboration program will help students appreciate and respond positively to the global nature of e-learning.

The lack of specificity was intentional: the initial ten-week program was viewed by both institutions as an exploratory step. Identifying explicit and measurable outcomes in advance was premature.

The actual forms of online collaboration to be employed, however, were clearly described in the Memo, as summarized below:

1) Threaded Discussion: A moderated discussion of current events in distance education will take place over a 4-6 week period. Utah State University will host the discussion on its WebCT server and will supply the moderator.
2) Early in the overall collaboration, there will be a text-based instructional chat. This will be a synchronous event, taking place in the late afternoon at USU and the next morning at SCNU.
3) The distance learning collaboration will conclude with an email-based game in which participants speculate on various scenarios about the future of online learning.

These activities comprised the initial collaboration. An important characteristic they all share is automatic documentation – transcripts and other archived records generated by the systems that support chats, discussions and email lists. Thus, the entire experience was captured and is available for research and analysis, maximizing the learning potential.

The Discussion

The online course management system most widely used and readily supported at USU is WebCT. It contains a threaded discussion tool, which was easily accessible to the SCNU students after passwords were issued to them. Both groups of USU Distance Education students were already using the same discussion forum.

The USU students were asked to post brief introductions, and to conclude with a question for the SCNU students. The goal was to create a forum for asynchronous interaction in which participants would learn something about each other, as well as explore issues of education and technology. It also created a welcoming environment for the SCNU students in that it was clear the American students were expecting them and looking forward to the dialogue.

As soon as the SCNU students were able to proceed, they joined the discussion in a flurry of responses and original posts. The following excerpted sample illustrates how the USU introductory statements began to generate interaction (participants used their real names in the discussions and chats, but they will be identified by program affiliation here):

Message No.854: posted by [ USU Student1], Fri, Mar. 2, 2001
Subject: Introduction
I am from a beautiful Southwestern French city, Biarritz. It is by Spain, on the Atlantic Ocean, by the mountains called “the Pyrenees” and the green countryside of the Basque Country. What I am doing in Utah? I am a student in the Instructional Technology department at USU. I’ve been in the program since last September, and I am glad to be there. My turn to ask you a question: Here in the U.S, the relationship between professors/students is extremely friendly especially at the graduate level. In fact, it is more a colleague/colleague type of relation. Do you experience the same type of relation with your professors in China?

Message No. 884: [Branch from no. 854]
posted by [ SCNU Student1] on Wed, Mar. 14, 2001, 07:59
Subject: Re: Introduction
Hello! I am interested in your question. As I know, some professors, especially famous specialists, in China are kind and equal to their students. They encourage the students to express their different views from themselves because sometimes students hesitate to disagree with professors who have a lot of prestige. I have attended a meeting on evaluating classes recently. Mr. L, a famous professor on instructional technology in China, and US took different opinions. He valued our opinion and decided to check the video of that class again. (I wonder if I expressed myself clearly, due to my poor English and some difference between American culture and Chinese culture.)

Message No. 886: [Branch from no. 884]
posted by [ USU Student2] on Wed, Mar. 14, 2001, 09:03
Subject: Re: Introduction
It is nice to hear from you and you are doing wonderful at expressing yourself in English! In your comments you stated that students hesitate to disagree with their professors. Is this because of your social beliefs or personal feelings? For example, I myself do not openly disagree with my professors. The reasons that I don’t disagree are because I don’t want to get a bad grade or I lack the knowledge or experience to back up my views.

Message No. 894: [Branch from no. 886]
posted by [ SCNU Student1] on Wed, Mar. 14, 2001, 21:50
Subject: Re: Introduction
I am excited to hear from you and am grateful for your encouragement. I rarely give my opposite opinion in public, mostly because I don’t want to break the harmonious tie with a little divarication. Provided our difference are not against my ethical belief, I prefer making full preparation and find another chance to tell him. As what you had said, we have no enough evidence, indeed, to back up our views on most occasions.

Over 150 messages were posted in the WebCT-based discussion during the first two weeks of interaction. For the American students, the link between participation and grading was never made explicit—when the courses were planned, an international collaboration was not even envisioned. In the USU-SCNU program, the opportunity to interact directly with Chinese students who shared a common interest in instructional technology led to more frequent and enthusiastic posts by the American students.

In addition to responding to the questions they found waiting for them, the SCNU students introduced themselves and posted questions of their own:

Message No. 874: posted by [ SCNU Student1] Wed, Mar. 14, 2001, 05:34
Subject: introduction of a Chinese girl in SCNU
I was born in a town in north China, which is famous for kites and where there is a international kite festival every year. Although my hometown is not as rich as Guangzhou, the city where I am now, I would love her forever. I came to SCNU last autumn after I graduated from Shandong normal university last summer. There is about 2,000 kilometers between home and the school. So it is too long for me to usually go home.I major in instructional technology and am doing research in educational TV, a unfashionable field. I don’t know how to deal with the relationship between technology and theory since we have limited time for study.
In addition, I am eager to become a teacher because I like children very much. And I want to know your opinions on teacher’s role in class.

Message No. 905: [Branch from no. 874]
posted by [ DLE Student1] on Fri, Mar. 16, 2001, 17:49
Subject: Re: introduction of a Chinese girl in SCNU
Hello L!
I was interested in your introduction. My daughter just began attending the University of Utah and also chooses not to come home because of distance. She loves the University though, and is studying Theater Arts. I would like to answer the question concerning the role of teachers in the classroom. Traditionally, teachers have been the resource of information, generally giving forth that information, and waiting for students to respond. My feeling is that teacher’s have a newer role in the classroom in this new technology era. They must be the guides, the leaders to a newer way. Many older teachers are still afraid of computers, and have a very difficult time getting accustomed to using new ideas to help them. They believe that tradition is still the better way. Perhaps they are afraid to fail at new ways.
I have a question back for you....did you choose to train to become a teacher because of prestige (importance of status) or because of your love for children? Is teaching in China considered high prestige? Most teachers in American choose teaching because they want to help children become better people, and in showing them the best ways. Is that also true in China?

Message No. 907: [Branch from no. 905]
posted by [ SCNU Student1] on Sun, Mar. 18, 2001, 06:00
Subject: Re: introduction of a Chinese girl in SCNU
Hello Mrs. P!
I’m pleased to hear from you and knew your daughter would be a teacher. I chose to train to become a teacher because of my love for children but not for prestige. Teaching in China is not considered high prestige. It is mostly because the smaller tuition fee that many poor students (is the word “poor” equal to the phrase “of humble origin?”) choose the normal universities. Another reason for students to be interested in teaching is that the salary of a teacher is ensured, though it is not so high. When I fill in the form about which university would I select, my parents didn’t agree with me very well. I insisted my idea because my teachers’ influence. They are all learned and respectable people and I hope to devote my life to education as they had been doing.
I have always argued that a computer or even the internet can not do as good as a teacher can do in cultivating the children. However, if the internet is able to give students enough knowledge, which way should a teacher choose to take effect on his students’ minds? Chatting with them through internet to tell them how to be a real human being, do everything excellently to set examples for them whether inside or outside the classroom, or give the ethic lessons on internet? The way of attending compulsory courses of elementary schools or middle schools through internet is new to most Chinese pupils. Is there any successful practice in U.S.? thank you.

The rapidly expanding discussion soon became difficult to follow. WebCT’s discussion tool, like most others, identifies the “Unread” messages for each logged-in participant, but the real problem was too many threads, too much going on. A sampling of the topics simultaneously addressed includes:

The Memo of Understanding envisioned, essentially, one big discussion and one big chat. It was obvious in a matter of days that a set of small groups was needed. People would then be able to focus on the lesser number of threads generated within their group, but also be free to browse other discussions if they wished to.

Message No. 906:
posted by ( USU Instructor) on Fri, Mar. 16, 2001
Subject: A Summary of the discussion so far....
I have been thinking more about the Discussion and Chat, and I believe that we will be better off starting with smaller group sessions. All of us at once may be too many people for the beginning.So, I have made up four groups—with one SCNU student, and some Instructional Technology Masters students and Distance Learning Endorsement teachers in each one.
I have also set up four discussion forums here, named Group 1, Group 2 etc. I would like us all to move the discussions into the Group Forums.
The first Chats will be done in these smaller groups also. Please use your Group Discussion Forum to schedule your Chat.
You are, of course, free to Chat about anything. But there will be a specific Task to accomplish. It will involve learning about differences and similarities in how we are trying to bring technology to the classroom in our countries. You will get more specific instructions about this soon.

International Text Chat

Using the small group discussion forums in WebCT to plan and schedule small group chats became a major project in itself. Synchronous events are difficult in the best of circumstances, even without 15-hour time differences and firewalls. With those and other complications, fostering real-time interaction was an extremely difficult undertaking.

The transcripts reveal a total of nine chats during a four-week period beginning in late March. Every one of them ended with the participants stating how much they had enjoyed the experience. However, these good feelings came at a cost – the time and effort in scheduling, and the feelings of frustration engendered when attempts to join a chat were not successful:

Message No. 967:
posted by [ SCNU Student2] on Fri, Mar. 23, 2001, 18:44
Subject: Re: What’s the matter?
I was rejected by the Text Chat Center again.Every Chat Room I wanted to enter told me the same message that the chat sever was not running.I feel unhappy.I will ask some specialists to help me deal with this.See you the next time!

Message No. 972: posted by [ SCNU Student3] on Sun, Mar. 25
Subject: SOS
I can not enter the chatting.The conversation text box tells me “connection is broken” Who can tell me how to do?It is time to begin .I am anxious. What about you all? Are you ok? Chatting?
Only two of the SCNU students were able to log onto a chat session, and they did so by using a dial-up connection in a private residence, rather than the computers at SCNU, which went through a proxy server. The SCNU server’s port 3000 was open for connecting to WebCT, but Chat is typically run through a different port, which was not open. Once this was determined, efforts were made to correct the situation:

Message No. 991: [Branch from no. 990]
posted by [SCNU Student2] on Tue, Mar. 27, 2001, 16:46
Subject: Re: Information on the chat
It’s of no surprise for me to find that I was rejected again this time.But I can’t help feeling regret.I have told my tutor the problems we have met. And he said he would do his best to get the Port 4446 opened.That will be the best for we can only log on USU Webct through the proxy and can not get one computer with an IP now.So I can do nothing for our text-chat now but asking for my tutor to give us good news as quickly as he can.I look forward the next time.I am anxious to think that you are having a hot discussion. How wistful I am to join you!

With prior testing and planning, perhaps the proxy server problem could have been addressed more successfully. As a pilot project, conceived and implemented on the fly, it is understandable that such technical difficulties would emerge. But understandability does not make the experience any less frustrating.

What, then, is the perceived benefit of a chat that justifies such a cost? In this case, coming as it did in the very early stages of a program, the attraction was clearly the chance to form personal relationships through real-time interaction. While a series of back-and-forth exchanges may play out over a number of days in an online discussion forum, in a chat the same exchange can be accomplished in minutes.

Each of the four groups had an assigned topic (e.g., Use of Collaborative Technologies, Internet Accessibility etc.), which they were to explore in a compare-and-contrast style. During the actual chats, which were moderated, some of the interaction focused on these topics, but not the majority of it. More than anything else, people shared information about themselves—their families, studies, careers, hopes, and dreams.

One of the four groups made the following points in their summary of the chat experience:

1. Getting to know people was the very best part
2. Everyone enjoyed the humor and the questions
3. Assigned subjects could not be discussed in depth because of time
4. The discussion forum is best for giving more thoughtful answers to questions

Despite the lack of depth, the chat transcripts show that a significant amount of information was transmitted, and not always of a purely personal nature. One chat focused on Internet access throughout the different levels of public education in the US and China, sharing observations about firewalls, censorship, costs and other barriers. In another, questions of private vs. public secondary schools in China and the US were considered.

The spy plane incident was not a major topic of the chats, but it was not ignored:

Time: Tue Apr 10 16:19:19 2001

S [ SCNU Student4]>>I think we needn’t see the political affairs would affect other things like that in acdemic domain./
T—MODERATOR [ DLE Student2]>>I agree, I think that is government business and we are real people. M, what do you think?
M [ USU Student3]>>I’m actually surprised that S would say that politics don’t affect the academic situation in China. I don’t know much about it, but I would’ve thought that the chinese government would see it differently./
T—MODERATOR [ DLE Student2]>>How do you think the academic situation in China will be affected by this political issue?
S [ SCNU Student4]>>If it has direct influence , then both sides of USU-SCNU students should hate each other. Is it right//
M [ USU Student3]>>I don’t mean to say that it seems one side would hate the other. I only know the US system and don’t think it will change, but I really don’t know for china. Obviously./
T—MODERATOR [ DLE Student2]>>We are just people, and I have found in my travels, that people have a lot more in common than not in this world, they love their families, have a desire to be successful in life, and many many more things like that. It is the govt that fouls things up. D, what about you>?
D [ DLE Student3]>>I think we are all in search of happiness as the Dali Lama says. We all want what makes us happy. This collaboration is good. I was very impressed w/the SCNU student interest and genuine kindness.
T—MODERATOR [ DLE Student2]>>I too am impressed with the collaboration and the kindness. I think it is good to talk.
S [ SCNU Student4]>>The people are always searching for happiness, but what the govenments search for is only “success’./
D [ DLE Student3]>>Or Power?/
T—MODERATOR [ DLE Student2]>>or Power!/

The “Depolarizer” Activity: An online game

Depolarizer is an email-based game developed by Sivasailam Thiagarajan, an innovative online educator commonly known as “Thiagi.” The purpose of the game is to share opinions on a controversial topic, think about how others may view the topic, and gain a greater understanding of other perspectives by role-playing an assigned position. At his website, Thiagi (1999) provides a ready-to-use template – all that is required is to select a topic.

On Monday, April 2, the last of the planned activities in the USU-SCNU Collaboration was launched, just as the spy plane incident began to emerge as a major diplomatic confrontation between the two nations. The email-based Depolarizer game consists of multiple rounds designed to exercise a range of different cognitive skills. It lasted for almost three weeks, while anti-American protests raged in China and President Bush sought the return of the crew and the aircraft without appearing to give in to Chinese demands for a formal apology.

As details of the international incident were taking shape, the Utah and SCNU participants were also receiving the following instructions:

There are Five Rounds to this game
This is the First Round:
Active collaboration between many Chinese students and American students in K-12 and Higher Education within the next five years
1. Extremely pessimistic
2. Highly pessimistic
3. Pessimistic
4. Slightly pessimistic
5. Neutral
6. Slightly optimistic
7. Optimistic
8. Highly optimistic
9. Extremely optimistic
Reply to this email and send me the number that indicates your personal position on the rating scale. Please add several statements explaining why you feel that way. I will know your rating and your explanation, but no one else will. I will not save the replies in a way that links them back to you.
Reply to me by Midnight, Thursday, April 5, 2001
I will email instructions for Round Two at 12:01 Friday, April 6
Things don’t always go as expected in first-time projects like this, but I am so glad to see that everyone is staying enthusiastic and doing their best. Thanks again.

Round Two involved prediction – the task was to estimate the average score from everyone who submitted a position in Round One. It is an exercise, which causes individuals to focus on how they think they differ from the norm. Once again, a brief explanation for the predicted mean score was required. This round had a winner. The person whose prediction came closest to the actual mean value was announced, and awarded a token prize of USU merchandise.

Next came the role-playing. On the topic of active collaboration between Chinese and American students, everyone was assigned a position – either Extremely Pessimistic or Extremely Optimistic – and asked to write three or more brief statements supporting that position. This was a very liberating activity. People were able to make strong statements without fear of attribution – they were just playing a role.

The written pieces submitted in Round Three were organized into an alternating series of Pro and Con arguments and emailed back to all participants. In Round Four, the initial question was posed again, after the experience of Rounds Two and Three as well as reading the full set of “Extreme” comments. Everyone was asked to indicate his or her current position on the one to nine scale. Round Five, the last round, was a reconsideration of the group’s attitude, accomplished by predicting the new group Mean.

Although there are similarities between this exercise and the traditional Delphi research technique, there are also important differences. The purpose for which Delphi was developed was to determine consensus; that is, to formulate a “best prediction” based on an iterative process involving a substantial community of experts. According to Witkin and Altschuld (1995):

During the iterative process, the needs assessor ascertains the degree to which individual respondents agree or disagree with the results from the entire panel, item-by-item, as well as their reasons for disagreement. Respondents consider, review, and rate ideas over an extended time period . . . (p. 194).

Thus, Delphi uses community as an instrument for achieving accuracy. Depolarizer has no such objective – it uses community instead to help individual participants focus on their own beliefs through a variety of distinctly different activities. The expected outcome of a Delphi procedure is a more accurate understanding of some important aspect of the future. The expected outcome of Thiagi’s Depolarizer game is a deeper understanding by the participants of their own thoughts and feelings about some important aspect of the present or the future.

Did Depolarizer change anyone’s perspective? Since no effort was made to impose a true experimental research design on the Depolarizer game, it is impossible to make causal inferences. Furthermore, the spy plane incident probably qualifies as the epitome of what Cambell and Stanley (1966) labeled as an “history” effect, threatening the external validity of research results as an influential outside event not anticipated by the research plan.

Nevertheless, since it was an inherently data-oriented activity, it is a simple matter to summarize the game with some descriptive statistics. The results of the four rounds in which participants submitted numbers, broken down by the sub-groups in the collaboration, were as follows:

Table 1. Depolarizer results


The results show a slight depolarization, as well as an overall increase in optimism. The highs and lows came closer. The gap between personal positions and predicted norms was almost eliminated.

The sub-group, which showed the greatest change in personal position (Round One – Round Four), was Females with a 14 percent increase in optimism. At first Females were less optimistic than males. By Round Four, the situation was reversed.

It is also striking that in Rounds One and Two, overall optimism was highest among the students (Chinese, then Americans) and lowest among the in-service teachers. During the experience, the DL Teachers made 12 percent increase in optimism, narrowing the gap considerably.

Finally, a brief review of the statements submitted in support of positions and predictions testify powerfully to the high degree of subjectivity involved in interpreting personal communications. The following excerpts are drawn from the Round Five submissions, illustrating clearly that different individuals can experience the same input in very different ways:

I found the optimistic position statements quite persuasive and I think the group may react the same way, thus the average should move toward the more optimistic scores ...

Realistically I feel this is still a pessimistic endeavor because of several of the barriers mentioned ...

I was more inclined to be more optimistic after reading the more personal responses. I think everyone else will feel more optimistic ...

The negative arguments were somewhat stronger than the positive ones ...

The optimistic student comments seemed more believable and made more sense. Although some of the barriers were valid arguments, I disagreed with a lot of the pessimistic comments


The first step in the USU-SCNU Collaboration featured relatively brief experiments with a varied set of activities – synchronous and asynchronous, moderated and unmoderated, one group and small sub-groups. There was much to learn from the successful aspects, and even more from the parts that did not work well.

In terms of a smoothly flowing activity in which participation was consistently high, the Depolarizer game was the most successful. This suggests that very tightly moderated activities composed of a series of small, assignable steps should be designed and utilized in future phases of the collaboration. As an asynchronous experience, the game was more engaging than the discussion.

The small group discussions never really had a chance to develop as a vehicle for deeper and more thoughtful interaction. As soon as the groups were established, they became chat-scheduling places, rather than true discussion forums. In the next iteration, the small groups should be in place from the outset, with a reasonable amount of time allotted for a shared project, such as reading and critiquing articles together.

Chats will always present more complications, as a real-time activity. They should not be allowed to become too much of a preoccupation, as they were during this phase. Alternative chat options should be investigated, such as ICQ or Instant Messenger services. Once the technical hurdles are overcome, they should be used on an occasional basis, perhaps with a pre-set schedule, to strengthen the human connections.

Finally, it is worth noting that cultural differences produced no discernable difficulties. The problems flowed from lack of experience in the appropriate instructional designs for the specific situation, and from technical limitations. During a period of time spiked with high emotions and nationalism, the participants in the USU-SCNU Collaboration communicated clearly and let their personalities show. The eagerness to make contact was apparent, and can be seen in the sometimes-lengthy excerpts included in this article.

Exploring the effect of participation in a diverse online community on the learning process will be the unifying theme behind the future research agenda. After a second semester (Fall, 2001) of similarly exploratory text-based interaction, plans have been made to extend the collaboration and to focus on the unique research opportunities that are presented. In specific, three areas have been identified for future activities:

1. A regularly scheduled series of videoconferences: Faculty and graduate students from USU and SCNU will make presentations as part of a regularly scheduled series, using a tested system operating at a negligible cost on the public Internet. Design elements involved in each event will be systematically evaluated by the all participants, with comparisons between Chinese and American perspectives being a feature of the analysis.
2. Discussion forums linked to the videoconference topics: An active on-going discussion provides the basis for experimenting with formats (List vs. Web-based), with moderator behavior, and with the linkage to a live event – always through the two different lenses.
3. Support for joint research and co-authoring: Working relationships among SCNU and USU faculty and students will be encouraged, particularly combined presentations and research activities.

Research will be both qualitative and quantitative. The different research traditions, or “different ways of knowing,” can complement each other very effectively in the broad new territory the collaboration enables. Is diversity an asset, and if so, how, and in what ways? How can technology be used most effectively to create diverse learning communities?

In the initial phase, the differences were felt to be an asset. In even the most tentative international online learning community, this important concept becomes very obvious simply from the increased levels of participation. It is a valuable lesson to be learned from a collaboration of this sort.

One of the Masters students from Utah State University expressed the inherent power of a diverse group of participants in these words:

I think it is the cultural differences that people are interested in and want to learn about. I see this as a motivator instead of an inhibitor.

Online learning communities can lead the way in modeling formats and activities that are strengthened by diversity. It might be what the world needs most.


Campbell, D. T., and Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Boston, MA.: Houghton-Mifflin.

Thiagarajan, S. (1999). Depolarizer: How to conduct this email game. Retrieved November 13, 2001 from: http://www.thiagi.com/email-depolarizer.html

Witkin, B. R, and Altschuld, J.W. (1995). Planning and Conducting Needs Assessments: A practical guide. Thousand Oaks, CA.: SAGE Publications.