Volume 9 Number 1
Boulder, Colorado
Fall 1994
Thomas Stewart, Center for Policy Research, SUNY_Albany
I am continuing to investigate the relations among seven components of judgmental skill: environmental predictability, fidelity of the information system, reliability of information acquisition, reliability of information processing, match between environment and judge, conditional (regression) bias, and unconditional (base rate) bias. This work includes both empirical studies and simulation.
One set of simulations focuses on G, examining and extending Castellan's 1992 results. These results have implications for anyone who might use G as a dependent variable in research. Another set of simulations examines how characteristics of the environment and the judge affect components of skill. For example, in one series of simulations with a three-cue environment, an equal-weight linear judge does poorly against a highly configural model (the multiplicative model with zero cue means) when the cues are uncorrelated. With moderate cue intercorrelations, the linear model provides a good approximation to the configural environment. G jumps to near 1.0 with even a small intercorrelation between the cues. Fidelity of the information system depends on measurement error in the cues and is nearly independent of cue intercorrelation. When the environmental model and the judgment model are fixed, properties of the environment such as cue intercorrelations and error in the cues have a substantial effect on performance. Of course, this is not news to Brunswikians.
The empirical study, conducted in collaboration with Paul Roebber of the University of Wisconsin/Milwaukee and Lance Bosart of Albany's atmospheric science department, involves the analysis of retrospective data on temperature and precipitation forecasts made by four human forecasters and a numerical model. The data set is unique because we were able to recover not only the forecasts and the actual weather events, but also the cues used to make the forecasts, so it was possible to analyze some of the skill components. We found that most of the skill in the forecasts could be accounted for by linear model. G was very high and not sensitive to weights. There was nonlinear variance that was common to several forecasters, but it did not contribute to skill. Humans did somewhat better than the numerical model. The importance of the "human element" in forecasting depends on how the forecast is to be used. Forecasters showed very little bias. The components of skill of the averaged forecasts closely resembled those of the best forecaster.
In addition, George Richardson, David Andersen, and I completed an experiment in dynamic decision making this year. The results suggest that knowledge of the deep structure of a dynamic system (a simulation of a social program) was of little help in managing the system . Information about management strategies was more useful. The results favor "operator logic" over "design logic" in the management of dynamic systems.
Ray Cooksey, Department of Marketing & Management University of New England, Australia
I am about two-thirds of the way through writing a book entitled Judgment Analysis: Theory, Methods, and Applications. The book to be published next year by Academic Press is intended for academics and postgraduate researchers/practitioners who wish to learn more about the paradigm of Judgment Analysis. I start with a very comprehensive discussion of the theoretical basis of Judgment Analysis and compare it to ten other "mainstream" theoretical approaches to decision making. Various research designs (e.g., single and double systems, IPL and IPC designs) and strategies are reviewed as are specific procedures and guidelines for designing sound Judgment Analysis research which will satisfy Brunswik's criterion for representativeness. Multiple regression methods are thoroughly reviewed as is the lens model equation from its early form to my recent derivation of the multivariate form and the Brier score decomposition form derived by Tom Stewart and his associates. I also discuss policy aggregation methods (e.g., ANOVA, cluster analysis). One chapter is devoted to a discussion of special topics in judgment analysis research (such as how to cope with categorical judgments and/or cues; how to handle unknown cue structures). The final chapter focuses on new directions for Judgment Analysis research including developments to make the approach more dynamic (including discussion of the potential utility of fuzzy set theory and chaos theory), the utility of statistical bootstrapping for hypothesis testing in Judgment Analysis research, and Judgment Analysis using personalized (idiographic) judgment ecologies. Illustrative examples and applications across a range of disciplines are liberally referred to in order to anchor particular ideas or methods. My hope is that I will have successfully pulled together, in one source, the essence of our research paradigm which, historically, has had aspects sprinkled throughout a very diverse literature.
Apart from my work on the Judgment Analysis book, I am also working on an expansion of Brunswik's probabilistic functionalism to encompass a more dynamic perspective. It is my preliminary thinking in this area which I will discuss at the 10th Annual Meeting of the Brunswik Society. Basically, I am using concepts from fuzzy set theory and chaos theory to elaborate and broaden Brunswikian concepts such as vicarious mediation and functioning, ecological and utilization validity, and the notion of judgmental consistency. My hope is that this work can begin to liberate us from our forty-year dependence upon multiple regression technology which has imposed a relatively static perspective on our work. By the time of the November meeting, I hope to have some empirical data to share which will explore the directions that fuzzy logic and chaos theory can move us (the data will also address the issue of time pressure influences on judgment within a standard judgment analysis task).
Roy Poses, Division of General Internal Medicine, Memorial Hospital of Rhode Island
Wally R. Smith, Donna K. McClish, Donna Alexander-Forti, several other investigators, and I have continued our study on Predictions and Outcomes of Congestive Heart Failure. Some interesting results concern the effect of stress on judgmental quality. Many believe that moderate stress could improve judgmental quality by getting people to focus on the most important cues, but extreme stress could decrease judgmental quality by distracting people from even essential cues.
We elicited emergency room physicians' judgments of the probability of several important outcomes of acute congestive heart failure (medical catastrophe within seven days, survival through 90 days, survival without major disability through 90 days). Simultaneously, we elicited information that might reflect the stress they were under at the time they made these judgments (at the time you saw this patient, how busy were you, how physically tired were you, how many hours had you slept in the last 24, [what has been] the number of admissions to the hospital?) We measured the quality of the physicians' judgments in terms of calibration and discrimination. We compared the aggregate quality of judgments made in high and low stress situations.
In general, the quality of the judgments was poor. Stratifying the judgments by the values of the individual stress variables, or by combinations of these variables, showed few clear differences.
A number of hypotheses may explain these results. Our measures of stress may have been inadequate. There may have been too little variation in stress among physicians to change their judgmental quality. There may have been a "floor effect," that is, the physicians may have had no idea about the important cues for this task in the first place, so stress could not have meaningfully affected their use of cues. Finally, the theory may not describe the effect of stress on these particular judgments. We look forward to discussing this work further at the Annual Meeting.
Frank Laufer, Rockefeller College of Public Affairs & Policy, SUNY_Albany
As part of dissertation research (which is only broadly defined at this point as examining the applicability of decision conferencing to consensus development by experts on controversial biomedical topics) a decision conference was recently conducted to determine factors to be considered when evaluating candidates for kidney transplant. During the first segment of the conference, the group's facilitator proceeded to elicit from the expert panel (comprised of a nephrologist, a surgeon, an ethicist, a nurse, a social worker, the local organ procurement officer, and a health policy expert) eight factors to be considered: medical prognosis, age, patient preference/motivation, probability of compliance, support and coping skills, history of substance abuse, social worth, and equity. Using social judgment analysis through POLICY-PC, and rating suitability of recipient on a scale of zero (not suitable) to four (highly suitable), two models based on organ source were determined. For cadaveric organs, a six-cue model was derived after the panel decided that history of substance abuse and support and coping skills should fall under compliance. The cues (with approximate relative weights), were medical prognosis (39%), patient preference (15%), probability of compliance (29%), age (14%), social worth (3%), and equity (1%). For organs from living-related donors, a four-cue model was determined: medical prognosis (25%), probability of compliance (40%), patient preference (25%), and age (10%). Informal follow-up to this point reflects a positive attitude towards the process.
Jeryl Mumpower, Center for Policy Research, SUNY_Albany
I am presently working actively on three projects with a distinct Brunswikian flavor. (Inactively, I'm working on about a half-dozen additional ones.)
First, I am continuing to work in the field of negotiations. I am just completing data analysis for two studies on interpersonal learning in negotiations (conducted in collaboration with Jim Sheffield, Tom Darling, and Rick Milter). Preliminarily, the data suggest that (a) people are not as severely affected by the "fixed-pie" bias as is widely supposed; (b) people don't typically achieve high levels of IPL in negotiations; and (c) good IPL doesn't seem to help negotiators very much to improve their individual payoffs.
Second, Tom Stewart and I are part of a team that is working on a descriptive study of clinicians' decision making in psychiatric emergency rooms concerning whether to admit potential patients.
Third, this same team is attempting to develop a computerized aid that will improve crisis decision making in psychiatric emergency rooms. Lens model ideas are prominent in both studies, although liberally combined with if-then branching rules. This type of decision making is quintessentially quasirational.
James Holzworth, Department of Psychology, University of Connecticut, Storrs
My colleague, Janet Barnes-Farrell, and I, along with several graduate students (Cheryl Ross, Martha Hennen, and Kristen Haggis) have conducted a series of judgment studies over the past year concerning performance appraisal. This project has been concerned specifically with corrective actions considered appropriate when addressing poor performance.
In our first study, student judges examined scenarios of poor performance in university jobs varying in terms of nine cues. Each judge considered, and then recommended (rated), the appropriateness of nine different corrective actions for each scenario. After making all nine ratings for a scenario, a judge picked one of the nine actions as the preferred one. Not unexpectedly, results indicated there were individual differences in use of cues when making recommendations for a particular corrective action, and there were individual differences in use of cues across different corrective actions. Multidimensional scaling and cluster analyses provided further insight into what cues were relied upon when making recommendation choices.
In the first study, judges were not aware of the consequences of the poor performance to the organization and or affected individuals. We conducted a second study using the same scenarios, and included outcomes with two levels of severity. After reading a scenario, student judges rated six contextual cues. Results indicated that outcome information significantly affected perceptions of three of the contextual cues. An important implication of these findings is that context effects may lead to differential treatment of workers who exhibit the same problem behavior.
We are currently analyzing data from our third study. This time we have experienced university supervisors making judgments concerning how to deal with instances of poor performance by hypothetical subordinates (with or without knowledge of outcomes). We are finding that supervisors rely on different subsets of information for different corrective actions. Cluster analysis is being used to identify groups of actions that show similar patterns in the kinds of information that influence judgments of their appropriateness.
A problem with this research has been fairly low values of policy consistency indices for most judges across the multiple action recommendations. I look forward to describing this work at the Brunswik Society meeting.
Kevin Coulson, Manhattan, Kansas
I started using the lens model shortly before JAWS II, and used the methodology in my dissertation. I'm currently putting the finishing touches on an article that proposes using the Achievement Index as an objective measure of expertise that can/should be tailored to the domain in question.
Dick Joyce, The Royal Colledge of Surgeons, Ireland
Dick Joyce and his colleagues at the Royal College of Surgeons in Ireland are continuing research and publications with the SJT-based Schedule for the Evaluation of Individual Quality of Life. Current applications include gerontology and head injury. Other uses include alcoholism (with Hans-U Fisch at the University of Bern). The SEIQoL Users' Manual is available from Professor Ciaran O'Boyle, Department of Psychology, Royal College of Surgeons in Ireland, Mercer St, Dublin 2, Republic of Ireland (E-Mail: cboyle@irlearn.ucd.ie). (Use of other methods of multiple regression analysis than PCPOL referred to therein is, however, recommended.)
Tim Earle and George Cvetkovich, Western Washington University
For the past two years, we have concentrated our research efforts on an attempt to develop a useful understanding of how people make judgments of social trust. Traditional, rationalist accounts of social trust claim that it is based on judgments of competence and responsibility. We drop the rationalism and claim instead that social trust is based on judgments of "cultrual values." An individual, for example, would tend to trust institutions that, in her judgment, operate according to values that match (or are similar to) her own. These values vary over time and social contexts as well as among individuals and cultual groups. In addition, we describe cultural values as being communicated in a narrative as opposed to a paradigmatic or scientific mode.
Social trust is traditionally spoken of positively, as an antidote to social distrust. What is missed in these accounts is that social trust is normally a within-group phenomenon_contributing to, rather than resolving social conflicts. We make a distinction between "pluralistic" (within-group) and "cosmopolitan" (across-group) social trust. Only the latter, future-oriented form of social trust_modeled on the cosmopolitan narrative of science_is useful in the solution of social problems. Our ideas on these matters, along with a bit of supporting data, are set out in a manuscript_"Social Trust: Toward a Cosmopolitan Society"_which we hope will be published next year.
Lubomir Kostron (Department of Psychology, Masaryk University, Brno, Czech Republik) continues to introduce Brunswik's ideas to his students through translations into Czech of papers by Egon Brunswik, Kenneth Hammond, David Leary, John Rohrbaugh, and Tom Stewart. Support and encouragement for his endeavors have been provided by Hammond, Leary, Rohrbaugh, and Stewart.
HTML formatting by Yong-Jin Cha, Center for Policy Research, University at Albany.
Return to Brunswik Society Home Page