At a Teaching Negotiation Resource Center (TNRC) faculty pedagogy seminar, members of the PON faculty and negotiation community gathered to hear Gordon Kaufman (MIT Morris A. Adelman Professor of Management, Emeritus) speak about how he uses quantifiable data to plot student-learning trajectories. The conversation focused on the ongoing debate within the negotiation pedagogy community regarding the way we assess instructional outcomes.
Quantif-why-able?
PON co-founder Howard Raiffa’s 1982 book The Art & Science of Negotiation explored many ways of quantifying negotiation outcomes. Raiffa (joint HBS/HKS Frank P. Ramsey Professor Emeritus of Managerial Economics) was a game theorist, and later a decision analyst, whose analytical approach offered fresh perspectives on negotiation. Raiffa established an experimental laboratory where students participated in, and improved upon, negotiation outcomes. Raiffa asserted:
One may never be able to predict or simulate in a laboratory setting all the aspects of complex real world negotiation, but there is no question as to the value of applying decision-theoretic concepts.
Raiffa recognized the importance of the microeconomic concept known as the Pareto Efficiency Frontier—named for its creator, Italian polymath Vilfredo Pareto. Today, the Efficient Frontier (Fig. 1) is considered an important instructional tool in negotiation, especially when used in conjunction with quantifiable negotiation simulations, a.k.a. scorable games.
Figure 1. Demonstrates that any agreement charted inside the frontier (e.g., R) is sub-optimal, and leaves open the possibility for both parties to achieve additional net gain without diminishing the other’s gain (e.g., Q).
Gordon Kaufman, a student of Howard Raiffa’s, began his presentation by quoting George Box’s 1987 book Empirical Model-Building and Response Surfaces: “…all models are wrong, some are useful.” This epigrammatic line identifies the rickety proposition at the heart of scorable games. Specifically, in the mutual gains model of negotiation instruction—where intangibles, creativity, and soft skills matter hugely—quantitative exercises can be helpful, but not always especially meaningful. So, why should educators use them?
The Case for Scorable Simulations
Kaufman uses quantifiable simulations as a metaphoric table-flip, upending his students’ expectations about their own negotiation prowess. He finds that scorable simulations offer a reality check for those with over-inflated images of themselves as master negotiators, and as a source of inspiration those who undervalue their own abilities.
James Sebenius (HBS Gordon Donaldson Professor of Business Administration) sees great value in scorable games when they are used in a lab setting:
It’s really hard to do good social science on something you cannot a priori specify. To get good lab work… you have to tie things down, vary something and see what happens. If you didn’t tie it down to start with, you’re going to have a tough time.
Sebenius considers scorable simulations most valuable when they provide both an intellectual and an emotional learning experience. Students can “crystallize metaphors,” both visual and analytical, using the outcomes from quantifiable simulations. Through this process, they can build a framework that will make it easier for them to continue to learn from their own future negotiation experience.
Sebenius has seen students awaken to the fundamental core lesson about the existence of joint gains, even inside the tight confines of a measurable interaction. This realization often helps students appreciate why joint gains are possible.
Dissecting Data
Kessely Hong (HKS Lecturer in Public Policy) suggests that before educators plot student efficiency on a Pareto Frontier, they must first determine which metrics they will use to assess student performance. This poses the question: which part of a student’s performance should we be trying to measure? When negotiation performance is tied to grades, students are likely to pay attention to the aspects of a negotiation that will be measured (often money and other data that is easily quantified). They may be tempted to ignore other valuable aspects of negotiation, such as trust, relationships, and respect, which are not being directly measured.
Similarly, Bruce Patton (co-founder and Distinguished Fellow of the Harvard Negotiation Project and co-author of Getting to Yes and Difficult Conversations) is curious about the performance data educators care most about. Should student insight be measured? Analytical prowess? Good results? Ability to learn? Ultimately, Patton believes teachers should think really hard about what they plan to do with any data they collect.
Hong notes the usefulness of the Pareto Frontier as a concept, but cautions that when students are graded solely on the proximity of their outcome to the Pareto Frontier, they may be tempted to falsely reduce their negotiation to a math puzzle. If negotiations were primarily mathematical problems, negotiation teachers would focus exclusively on teaching mathematics, and number crunching analysts would be able to solve all high-stakes conflicts. We know that’s not the case.
Robert Bordone (HLS Thaddeus R. Beal Clinical Professor of Law, and founding Director of the Harvard Negotiation and Mediation Clinical Program) questions all the “hand-wringing” about metrics in the negotiation instructional community. He maintains that some negotiation educators seem more focused on the signs of incremental, measurable improvement than educators in other related fields. Bordone poignantly hypothesizes:
I sometimes wonder if our field’s worry about quantifying learning, and making it measurable by plotting it at the end of an experiment, suggests that we feel like we need to prove something to others outside our field. There are of course aspects of negotiation teaching that can be improved and research showing what is landing and what is not is important. At the same time, some of what we offer students isn’t easily measured and we should be confident, by dint of our own conversations, qualitative feedback mechanisms, and day-to-day observation, that our teaching is making a difference.
Disincentives of Data
The issue of whether to score participants in a negotiation exercise, and what to do with the data, are only part of the story. What if attempting to quantify student performance has a negative impact on student performance itself? What if an effort to reduce everything to numbers actually gets in the way of what we are trying to teach?
According to Hong, “knowing what’s being measured and not measured may result in over-emphasis on particular aspects of negotiation that are not especially important.” She offers the following rhetorical proposition:
What is the cost of experimenting by trying new negotiation strategies?
- In the real world? Money, reputation, career, etc.
- In class with outcome-based grades? Grades themselves.
- In class where grades are not directly tied to outcomes? Nothing.
She goes on to assert that if money, reputation, and career are on the line, participants are less likely to experiment with new theories or unfamiliar negotiation methods. They may be more likely to revert to potentially sub-optimal tactics that are familiar and appear less risky.
If grades are at stake, students are more likely to do what they can to maximize their grade. They may even be inclined to “game the game” (e.g., both sides reveal all information early and are unrealistically generous in pursuit of an agreement). Given that students often form close ties to their classmates, they may not be willing to commit fully to the “value claiming” aspect of negotiation for fear of damaging their classmates’ grades.
Conversely, when no grades, money, reputations, or careers are on the line, students may feel much more comfortable trying out new negotiation techniques. Hong reports that students who are not being graded or scored on the outcomes achieved in class negotiation role-play simulations are still heavily invested in their performance, due in part to the visibility of their performance in the eyes of their classmates.
Moving Beyond the Game…
In his teaching, Kaufman links scorable game results with each student’s counterpart’s evaluation of his or her intangible skills. Combining the two reports creates a fuller picture. It places a numerical score in the context of a negotiator’s overall conduct. For example, a report on a high-scoring negotiator might be accompanied by an evaluation that speaks of a disrespectful approach to a negotiating partner, poor communication skills, deceptive use of information, and an overly competitive attitude. The inverse could illustrate the opposite end of this potential continuum.
Hong takes this idea one step further and asks each negotiator to enumerate the different goals that they were trying to achieve during a negotiation. She then asks the negotiator’s counterpart to engage the student in a conversation about how well each thinks these goals were achieved. This allows negotiators to assess whether their counterparts correctly perceived their intent and their actions.
In order to evaluate whether a particular group has improved, Patton uses the popular “Arm Exercise” only after foundational negotiation concepts have been presented, rather than at the beginning of a class where it normally appears. The “improved” exercise results—more pairs earning points through collaboration—give him a good sense of how much learning has occurred in the room.
Lawrence Susskind (MIT Ford Professor of Urban and Environmental Planning and director of PON’s Pedagogy Initiative) encourages educators to use more video reflection in their classes. He uses video recordings, often prepared by the students themselves, extensively in his courses. By showing edited clips of specific points in a negotiation (e.g., “openings”, etc.), he tries to provoke discussions of a kind he is not otherwise able to achieve. The recorded interactions provide a faithful rendition of what happened, on both the formal and tacit levels. Each negotiator’s strategy and its impacts are obvious to the whole class. Video reflection allows Susskind to confront students more effectively and to compare how different pairs of students handled the same assignment. When students have to record, edit and comment on videos of their fellow students, they become much more self-conscious about their own personal theories of negotiation practice.
Patton agrees that this method of assessing a student’s performance, captured on camera, not only allows the student to gain an improved perspective on his or negotiation style, but also allows others to study key interactions in great detail. He envisions a future video “decoder” that will analyze and describe negotiating behavior, including at a micro level, as a potential game-changer in negotiation instruction.
Bordone’s self-stated goal as a teacher is “to increase students’ self-awareness.” To help him accomplish this, he requires his students to keep a weekly journal in which students accumulate instructor feedback. While this does little to provide empirical evidence of improvement, it does give Bordone increasing insight into his students’ growth and learning.
Lisle Baker (Professor of Law, Suffolk University) suggests that we focus more on how to measure negotiation learning within pairs and teams. Baker wonders if there aren’t lessons to be learned from sports like martial arts, tennis, etc., where individuals seek to improve their performance as they interact with others. Could there be drills, workouts, practices and rehearsals, beyond simulations, that hone a negotiator’s skills to near-instinctual levels?
Conclusion
Scorable simulations provide an opportunity to quantify learning levels, and by extension, successful teaching. Watching students get better at reaching the Pareto Frontier through the application of negotiation skills can give educators and students important information. Success in the quantifiable world, however, is only one measure of improvement. We need to invent better ways of merging quantitative assessment of negotiation performance (such as scorable games) with other tools and techniques that get at other dimensions of negotiation success.
Both Sebenius and Patton point to the Program on Negotiation’s Great Negotiator Award recipients as prime examples of expert negotiators who have mastered the full array of tools and techniques. Sebenius notes one commonality among these individuals: a refusal to accept the confines of the problem as it is initially presented to them. By adding/subtracting parties, reframing issues, and generally being creative with the structure and process of conversation and problem-solving, Great Negotiators are able to achieve unexpected and highly valuable results.
Simulations allow students to embrace unfamiliar situations (and thus build up their improvisational capabilities). Current research reported in the Negotiation Journal shows that learners grasp and retain concepts better in experiential learning environments, like simulations, in contrast to theoretical teaching methods, like lectures (Druckman & Ebner, 2013). So, while scorable simulations have a range of strengths and weaknesses, simulations – whether scorable or not – can contribute to negotiation instruction in a number of ways.
Reference List
Raiffa, Howard. The Art & Science of Negotiation. Cambridge: Harvard University Press, 1982.
Box, G. E., & Draper, N. R. (1987). Empirical model-building and response surfaces. New York: Wiley.
Kaufman, Gordon; Sebenius, James; Hong, Kessely; Patton, Bruce; Bordone, Robert; Baker, R Lisle; Susskind, Lawrence. “Tracking Student Learning Trajectories: Designing Experiments in Integrative Bargaining.” Lecture, TNRC Faculty Dinner Seminar, Harvard University, Cambridge, November 3, 2015.
Druckman, D., & Ebner, N. (2013). Games, Claims, and New Frames: Rethinking the Use of Simulation in Negotiation Education. Negotiation Journal, 29(1), 61-92. doi:10.1111/nejo.12005
