IECM 12.0 beta User Manual > Introduction to Uncertainty Analysis >

Designing an Elicitation Protocol

Studies of uncertainty judgment show that the most frequent problem encountered is overconfidence. Knowledge of how people make judgments about probability distributions can be used to design a procedure for eliciting these judgments. The appropriate procedure depends on the background of the expert and the quantity for which the judgment is being elicited. For example, if you have some prior knowledge about the shape of the distribution for the quantity, then it may be appropriate to ask you to think about extreme values of the distribution and then to draw the distribution yourself. On the other hand, if you have little statistical background, it may be more appropriate to ask you a series of questions. For example, you might be asked the probability of obtaining a value less than or equal to some value x, and then the question is repeated for a few other values of x. Your judgment can then be graphed by an elicitor, who would review the results of the elicitation with you to see if you are comfortable with your answers.

To overcome the typical problem of overconfidence, consider extreme high or low values before asking about central values of the distribution. In general, experts' judgments about uncertainties tend to improve when:

•The expert is forced to consider how things could turn out differently than expected (e.g., high and low extremes).

•The expert is asked to list reasons for obtaining various outcomes.

While the development of expert judgments may be flawed in some respects, it does permit a more robust analysis of uncertainties in a process when limited data are available. Furthermore, in many ways, the assessment of probability distributions is qualitatively no different than selecting single "best guess" values for use in a deterministic estimate. For example, a "best guess" value often represents a judgment about the single most likely value that one expects to obtain. The "best guess" value may be selected after considering several possible values. The types of heuristics and biases discussed above may play a similar role in selecting the value. Thus, even when only a single "best guess" number is used in an analysis, a seasoned engineer usually has at least a "sense" for "how good that number really is." This may be why engineers are usually able to make judgments about uncertainties, because they implicitly make these types of judgments routinely.