Factorial Experiments

I am interested in Dr. Collins’ work on optimizing behavioral interventions, and I was surprised that she advocates the use of factorial experimental designs for some experiments. I was taught that factorial experiments could never be sufficiently powered. Can factorial designs really be implemented in practice? — Signed, Fretting Loss of Power


Dear FLOP,

This is a common question. Experimental subjects are often expensive or scarce. In these cases, a factorial experiment can save money and resources and still provide sufficient statistical power.

Dr. Welton is revamping a high-risk teen drinking intervention to be delivered in mental health settings, and she is deciding which of the following components to add:  (A) peer-based sessions; (B) parent-based sessions; (C) drug resistance skills; and/or (D) drug use education. Dr. Welton wants to decide which combination of the new components will achieve the best participant adherence. Dr. Welton considers two different design approaches:

  1. Conduct four separate experiments, one corresponding to each of the intervention components.  Each experiment would involve two conditions: (i) a condition in which the intervention component is included, and (ii) a control condition in which the intervention component is not included. For each of the four experiments, the data analysis would produce an estimate of the mean difference between the two conditions.
  2. Conduct a factorial experiment. This approach would treat each of the four components as a factor (i.e., independent variable) that can take on the levels “not included” or “included.” Crossing these factors would result in 16 experimental conditions (every combination of the levels of the four factors) and provide estimates of the main effect of each component and all interactions between components.

Next, Dr. Welton considers the cost of each approach. She anticipates a per-subject cost of $100. In addition, the overhead associated with each experimental condition (except for the control condition) will be $500, which is the cost of developing materials and training staff for each new version of the intervention. She specifies that each component has to produce an effect of at least .35 of a standard deviation in order to be considered for inclusion in the new intervention. Therefore, Dr. Welton wants to be sure she has the power to detect effects of at least this size.

Table 1
Cost Comparison of Two Experimental Design Alternatives
Approach N needed to achieve power ≥ .81 Subject costs Number of experimental conditions Experimental condition costs Total costs
Separate experiments 1,088 $108,800 8 $2,000 $110,800
Factorial experiment 272 $27,200 16 $7,500 $34,700
1Power based on effect size corresponding to treatment/control difference in separate experiments, main effects in factorial experiment.

Table 1 compares the cost of each approach. To achieve the desired statistical power in this example, the separate experiments approach requires four times as many subjects as the factorial experiment. Dr. Welton would save $76,100 by conducting a factorial experiment instead of four separate experiments! Collins, Dziak, and Li (2009) discuss how to determine which of several design alternatives is most cost-effective in a given situation. A free SAS macro for comparing the relative costs of a variety of experimental designs can be found at http://methodology.psu.edu/downloads/sasmacro.