In the 60's early 70's "policy capturing" was universally used.  In the 70's and the 80's "judgment analysis" came into widespread use, but both terms were considered synonymous.  Now we have a clear view of the distinction between these two terms, and we can use the correct term when appropriate.  The following comments trace this evolution and describe the current view.

Following up on Ken Hammond's message about the origins of social judgment theory, and also in response to a private message from Len Dalgleish, I've been doing a little investigation of the terms "policy capturing" and "judgment analysis."

First, judgment analysis (the term I strongly prefer) is not synonomous with SJT.  Judgment analysis is just a method for modeling judgment. It is often used in SJT research, and it is useful in implementing some of the remedies suggested by SJT, but there is much more to SJT.

Both terms originated at about the same time, in the early 60's, in connection with work being done at Lackland AFB (see appended messages from Joe Ward and Jim Naylor). "Policy capturing" was first used by Christal and Bottenberg. Naylor and Wherry, who were doing contract work with Lackland used the acronym "JAN" (Judgment ANalysis) in 1965 to refer to a technique for clustering judges based on their judgment policies. Sam Houston (1974), also associated with Lackland, wrote a monograph with judgment analysis in the title. Classic papers in the Brunswikian tradition, such as Hammond (1955), Hoffman (1960), and Slovic and Lichtenstein (1973) do not use either term.

For a long time, I have been encouraging the use of the term "judgment analysis" instead of "policy capturing." I can remember the exact moment when my distaste for the term "policy capturing" peaked: I was sitting next to Len Adelman at a meeting when somebody mentioned policy capturing. I looked over at Len and he was doodling a stick figure carrying a butterfly net and chasing a "policy." I then realized that we don't capture anything, we analyze judgment. Furthermore only a few people (mostly psychologists) have used the word "policy" to mean "judgment policy." To everyone else, it has a different meaning. I am happy that Ray Cooksey has used "judgment analysis" in his forthcoming book.

Of course, "policy capturing" is still in widespread use. In bibliographic searches, one of the best ways to find studies that use judgment analysis is to search for the keywords "policy capturing." In the current issue of Medical Decision Making is an article with "policy capturing" in its title.  Among its authors are highly respected subscribers to this list who are well acquainted with judgment analysis. Presumably, they chose "policy capturing" for a reason. They may want to respond to this message.

One unfortunate terminological hybrid must be stopped. I have seen policy capturing and judgment analysis combined into "policy analysis." We really don't need a third term to refer to the same procedure. This one is obviously unacceptable since policy analysis has long been used to refer to a completely different type of inquiry. I hope we can put a stop to it before it spreads.

..Tom Stewart

References:

Hammond, K. R. (1955). Probabilistic functioning and the clinical method. Psychological Review, 62, 255-262.

Hoffman, P.J. (1960). The paramorphic representation of clinical judgment. Psychological Bulletin, 57, 116-131.

Houston, S. R. (1974). Judgment analysis: Tool for decision makers. New York: MSS Information Corporation.

Naylor, J. C., and Wherry, R. J., Sr. (1965). The use of simulated stimuli and the "JAN" technique to capture and cluster the policies of raters. Educational and Psychological Measurement, 25(4), 969-986.

Slovic, P., and S. Lichtenstein, (1973). Comparison of Bayesian and regression approaches to the study of information processing in judgment.  In L. Rappoport and D. A. Summers (Eds.), Human Judgment and Social Interaction. New York: Holt, Rinehart & Winston, pp. 16-108.

====================================================================
Comments from Joe Ward:

I have just talked with Bob Bottenberg about your questions. We both agree that THE ORIGIN OF TERMS "POLICY CAPTURING" and "JUDGMENT ANALYSIS (JAN)" as used at the Personnel Research Laboratory at Lackland AFB came from informal discussions among Ray Christal, Bob Bottenberg and Joe Ward. The terms may have been used earlier and may have been publicly documented earlier by others; however, Bob and I believe that we did not know of the use of these expressions by others.

The idea of "POLICY CAPTURING" using REGRESSION MODELS to "mathematically capture a policy" was stimulated from the recognition in the 1950's that the PERSONNEL ASSIGNMENT PROBLEM (assigning persons to jobs, or "person-job match") was mathematically the same as the TRANSPORTATION PROBLEM OF LINEAR PROGRAMMING.

When we talked about trying to "assign personnel to jobs to maximize "payoff" or "value" to the Air Force", the universal response was that "But we don't know the "payoff" or "value" of each person on each job"!  Then our response was "Then we do not need the many counselors who are trying to put each person into the "right" job!" "So we can just assign the personnel at random"!

Of course, the counselors probably were doing something toward "maximizing" the "payoff" of the assignment process. So we decided that we might be able to "capture" the counselors "policy" with one or more regression models and then we could fill in the "payoff" array with the predicted values from the regression model(s). Then I developed the DECISION INDEX to provide counselors with information that would allow them to approximate the "optimum" assignment of personnel to jobs. The DECISION INDEX is about the best approach to the "sequential" personnel assignment problem.

Later we incorporated the idea of "clustering" regression equations to determine how many different "policies" were involved.

Several of the original Air Force Publications have been published in the Journal of Experimental Education. This happened because Jack Schmid at Univ. of Northern Colorado at Greeley was editor of the J of Exp. Ed. Sam Houston did a lot of Policy Capturing while he was at Univ. of Northern Colorado, and Sam's wife did her dissertaion (at UNC) using Policy capturing in the study of Pornography.

The most concentrated source is in the J of Exp. Ed. vol. 36, No.4, Summer'68  But the J of Exp. Ed. contains other examples.

This volume contains:

1. JAN: A TECHNIQUE FOR ANALYZING GROUP JUDGMENT by Ray Christal (Footnote #2 indicates "Dr. Joe H. Ward, Jr. is credited with suggesting use of a least-squares-weighted regression formula to capture the policy of a single rater: also see (2)." The (2) is Paul Hoffman's article, The Paramorphic Representation of Clinical Judgments", Psychological Bulletin, LVII (1960), pp. 116-131.

2. SELECTING AN HAREM - AND OTHER APPLICATIONS OF THE POLICY-CAPTURING MODEL by Ray Christal.

3. AN ITERATIVE TECHNIQUE CLUSTERING CRITERIA WHICH RETAINS OPTIMUM PREDICTIVE EFFICIENCY.

JAN - which has been used to describe both the "POLICY CAPTURING" using regression models and the "CLUSTERING" of the POLICY EQUATIONS -combines regression modeling and "CLUSTERING" that was first defined in 1961 in HIERARCHICAL GROUPING TO MAXIMIZE PAYOFF and then revised and published in the J. of the ASA in 1963. This original hierarchical clustering algorithm is now contained in many of the statistical packages.

The DECISION INDEX was first described in THE COUNSELING ASSIGNMENT PROBLEM BY Ward in Psychometrika, 23, 55-65
********
For reference to the origin of POLICY SPECIFYING see J. of Exp. Ed. v.48,1
CREATING MATHEMATICAL MODELS OF JUDGMENT PROCESSES: FROM POLICY-CAPTURING TO POLICY-SPECIFYING by Joe Ward

Policy Specifying provides a procedure to create prediction models and allows the judge to observe the output of the models. Then the models are modified and the judge takes another look. This continues until the output of the function seems OK. This approach was developed for the Air Force PROMIS system for recruiting. The idea was created to allow for the judge to express more easily "INTERACTIONS" among variables. Our observations with "POLICY CAPTURING" is that it is not easy for judges to express interactions and non-linearities. But POLICY SPECIFYING MAKES THIS EASIER.

About Policy Capturing vs. Policy Specifying

Policy Capturing attempts to predict the judgments from a mathematical model.

Policy Specifying attempts to allow the judge to create a mathematical model that produces output values ("judgments") that are desired by the judge. Rather than make judgments and then try to allow the regression model to reproduce the judgments, the Policy Specifying approach allows the judge to define a model that hopefully produces output that expresses the judge's policy. This approach seems to allow for the expression of interactions and non-linearities.

-- Joe
(joeward@tenet.EDU)

====================================================================
Comments from Jim Naylor:

It gets a bit murky, particulary since I cannot locate a copy of the Bottenberg and Christal Tech Report from 1961.

Bottenberg, R. A. & Christal, R. E. An iterative technique for clustering criteria which retains optimum predictive efficiency. WADD-TN-61-30, AD-261 615. Lackland AFB, Tex.: Personnel Research Laboratory, Wright Air Development Division, March, 1961.

My memory is that the first ACTUAL use of the term "policy capturing" in print was not in the above, but in the tech report

Naylor, J. C., and Wherry, R. J. Feasibility of Distinguishing Supervisor's Policies in Evaluation of Subordinates by Using Ratings of Simulated Job Encumbents. PRL-TR-64-25. Personnel Reseaarch laboratory, Aerospace Medical Division, Air Force Systems Command, Lackland Air force Base, Texas, October, 1964.

I have a copy of the above and the term policy capturing is used numerous times.

The first journal article to use both the terms "policy capturing" and "Judgment Analysis" was the 1965 Naylor and Wherry paper in EPM.

The first ACTUAL use of the Judgment Analysis term was in the tech report of Christal:

Christal, R. E. JAN: A Technique for Analyzing Individual and Group Judgment. Lackland AFB, Tex.: PRL-TDR-63-3, AD-403-813. Personnel Research Laboratory, Aerospace Medical Division, February, 1963.

Confused! Hope this helps some.

Jim

James C. Naylor, Ph.D. 614-292-3038 Office
Chair, Department of Psychology 614-292-4537 Fax
The Ohio State University
142 Townshend Hall naylor.2@osu.edu

================================================================

Additional comment from Naylor:
Tom...

Looks fine to me. I don't disagree with any of your comments. I'm glad joe could confirm that the Bottenberg and Christal paper contains both terms. My own concern about using the term Judment Analysis to replace policy capturing is that I have always seen JAN as a very specific technicque developed at Lackland for clustering policies. The first stage of that process involved capturing the policies of individual raters using linear regression. To me these two terms are quite explicit in meaning and are NOT substitutable.
Jim

 >>>End of  Posting number 180, dated 1 Aug 1995 09:08:55

Policy capturing is simply the application of regression analysis to modeling judgment. It is not based on the lens model or any Brunswikian ideas. Policy capturing studies are atheoretical, generally do not include the environment side of the lens model, generally use orthogonal designs, and generally use linear regression (although ANOVA and conjoint measurement is also used), and often confront people with unfamiliar tasks.

"Judgment analysis" to refers to modeling of judgment in the Brunswik/Hammond tradition. This requires an analysis of the environment, which policy capturing does not include. Furthermore, judgment analysis is clearly not wedded to regression.
Judgment analysis requires studying both sides of the lens model and using a task that is familiar to subjects, as well as gathering data under representative conditions. Representative design is meaningless unless task is based on a naturally-occuring judgment problem.

    Policy capturing has traditionally only referred to the modeling of judgments and is almost always associated with multiple regression models. However, this is just a small aspect of judgment analysis which involves a full implementation of representative design, considerations of task ecology as well as judgment process, and need not be tied to multiple regression analysis. In the context of judgment analysis, 'policy capturing' in the Bottenberg, Ward & Naylor sense, is exclusively linked to the single system model which ignores the existence of an ecological criterion. However, it is still quite common for people who do 'judgment analysis' to refer to the process of modeling judgments as 'policy capturing'.

    Thus, there is a technical use of 'policy capturing' which refers to a stand-alone method linked only in a very minor way to Brunswik's ideas (if one uses the single system conceptualization) and a less formal sense which refers to the methodological exercise of modeling judgments in the larger context of at least a double system lens model. The former sense is synonymous with multiple regression modeling of judgments made on a series of profiles (usually hypothetical) whereas the latter sense potentially encompasses any statistical, mathematical or modeling procedure that allows one to model or 'capture' judgments as well as models of an ecological criterion (which then permits the measurement of achievement). My view is this latter sense is the one which continues to keep 'policy capturing' alive as a term of reference for what we do and is why the term is so hard to extinguish. I deliberately titled my book 'Judgment Analysis' to signal this but even I am guilty of slipping into using the term in its less formal sense (witness chapter 4 in my book, which is entitled 'Capturing Judgment Policies'!). The debate is further confused if we throw the label 'Social Judgment Theory' into the fray! This label is largely synonomous with Judgment Analysis as it has evolved - but both labels are still commonly used (witness an entire edition of Thinking & reasoning devoted to SJT). What I think we are seeing is the slow process of evolution at work, which for a period of time, means that many related 'species' may co-exist before one comes to predominate. Tom is arguing for Judgment Analysis to predominate and I agree - it signals best what we are doing.
 
    Linearity is only a part of judgment analysis by association with regression models which are most commonly linear in composition. However, nothing in Judgment Analysis technically requires an assumption of linearity and there are many examples of nonlinear judgment modeling in the literature. However, I signaled in Chapter 8 of my book that new modeling techniques need to be developed which are inherently nonlinear so as to better represent dynamic judgment systems (including models that admit fuzzy logic and chaotic dynamics). Such models are only now beginning to emerge. Whether or not they will do a better job than linear models remains to be seen - many would invoke the law of parsimony and say that if nonlinear models add only incrementally to what a linear model can predict, then stick with the linear model.
 
    My personal view, however, is that we have over-simplified our modeling systems for long enough and it is time to 'complexify' them to as to more appropriately encompass the constraints, contexts and conditions under which judgments are made. For this type of effort, multiple regression and related statistical models just will not do. In fact, we may not have the mathematical technology yet available to do such modeling (although recent developments in dynamic systems theory and modeling look promising), and this means that qualitative mapping approaches may be the first approach to attacking the problem (a perspective I am currently developing with respect to the judgments and decisions made by organisational managers and CEOs as well as magistrates and court judges and justices).

Associate Professor Ray W. Cooksey
Department of Marketing & Management
University of New England
Armidale, NSW 2351
AUSTRALIA
phone: +61 2 6773 2563
fax: +61 2 6773 3914
email: rcooksey@metz.une.edu.au
Web Site: http://metz.une.edu.au/~rcooksey