Decision Making Using Data From a Factorial Experiment: Practice Data Sets

About the Practice Decision-Making Data Sets

In the optimization phase of MOST the investigator conducts an optimization trial, which is an experiment to gather information needed in deciding which components and component levels will be selected for inclusion in the optimized intervention.

In general, intervention scientists do not have much experience with this kind of decision making. In our work, we have found it extremely helpful to practice the decision-making process using artificial data WELL BEFORE analyzing the empirical data. This provides a way of gaining a little experience in advance.

We have provided for your use two artificial data sets generated to resemble data from actual implementations of MOST. Both simulate factorial optimization trials.  Here is how we recommend using each data set:

  1. Conduct the factorial ANOVA on the data.
  2. Prepare plots of any interactions that are likely to be important in decision making.
  3. Hold a meeting of all decision makers, and go through the exercise of selecting components and component levels based on the factorial ANOVA results and plots.

NOTE: Be sure to allow enough time – the decisions may not be straightforward. At a minimum, allow two hours.

The data sets and accompanying materials are supplements to a journal article that describes one approach to decision making. You can use the approach described there, or any rational approach. If you plan to use an optimization criterion that involves cost, you can make up data on the cost to implement each component level, and/or the time it takes to implement each component level, and incorporate this information into the decision making process.

We have provided factorial ANOVA output and selected plots so you can check your results against ours. We also provide an “answer key,” which is simply a rank ordering of the experimental conditions by the true mean on the outcome variable. In real life this would be unknown, of course, but because these are artificial data we generated, in this case we know the true experimental condition means. We STRONGLY recommend that you do not look at the answer key until AFTER the decision making is complete.

Do not expect that your decision making process will lead you to the combination of components and levels that is ranked #1 in the answer key. The objective of MOST is NOT to identify the #1 ranked combination; rather, it is to enable the investigator to identify ONE OF THE TOP RANKED combinations, or, put another way, to weed out the poorly performing combinations. Moreover, the ranking in the answer key does not take cost into account. The combination of components that provides the best expected outcome without taking constraints into account is likely not to be the best when constraints are applied.  For example, it may be too expensive.


Collins, L. M., Trail, J. B., Kugler, K. C., Baker, T. B., Piper, M. E., & Mermelstein, R. J. (2014). Evaluating individual intervention components: making decisions based on the results of a factorial screening experiment. Translational behavioral medicine4(3), 238-251.


Last updated: May 5, 2020


A pediatric obesity researcher is interested in using a 25 factorial design to select the components/component levels for a family-based intervention in which a child is obese.

Open example 1

Example 2

A smoking cessation researcher is interested in using a 25 factorial design to select the components/component levels for a smoking cessation intervention.

Open example 2