Suggestions for Planning a Grant Proposal Involving MOST

Here we offer some informal suggestions intended to help people who want to propose to implement MOST in their work.

We assume that you have a basic familiarity with factorial and fractional factorial designs. If not, you may wish to read “An informal introduction to factorial experiments” aimed at those with a background in the RCT. You may also be interested in our “FAQ about factorial experiments.”

 

Offer evidence that MOST is better than the traditional approach in the long run

If your aim is to develop more effective and efficient behavioral interventions in a systematic fashion, it seems logical that MOST is a better long-run strategy than the traditional treatment package approach. But to convince the skeptics out there, hard evidence is necessary.

It is difficult to imagine conducting empirical research to compare MOST and the treatment package approach. One scenario would be randomly assigning, say, 100 behavioral scientists to use either MOST or the treatment package approach over a 10-year period, and then comparing the effectiveness and efficiency of the resulting interventions. That is obviously not practical.

Given that an empirical study to test the usefulness of MOST is out of the question, the next best thing is a statistical simulation. Some collaborators and I did an extensive statistical simulation to explore whether, when, and how MOST is better than the treatment package approach. You can read about it in

Collins, L. M., Chakraborty, B., Murphy, S. A., & Strecher, V. (2009). Comparison of a phased experimental approach and a single randomized clinical trial for developing multicomponent behavioral interventions. Clinical Trials, 6, 5-15. PMCID: PMC2711350

 

Describe MOST succinctly

MOST is a comprehensive framework, not an off-the-shelf procedure. I suggest not trying to cover the broader picture of MOST in your proposal. You have decided what aspects of it you are going to implement in your study, so confine your description to these parts of MOST. It may help to include a flow chart like the one in Collins et al. (2011), BUT one developed specifically for YOUR project. For example, if you are not going to do any refining, just leave the refining phase out of the flow chart. You may want to add a box representing pilot testing, if pilot testing is going to demand a significant amount of resources and time. Although figures can take up a lot of room, including a figure adds clarity and can actually save room if it means your narrative description can be shorter.

Reference

Collins, L. M., Kugler, K. C., & Gwadz, M. V. (2015). Optimization of multicomponent behavioral and biobehavioral interventions for the prevention and treatment of HIV/AIDS. AIDS Behav,Supp 1, 197-214. doi:10.1007/s10461-015-1145-4

 

Show that a factorial experiment will be more economical than the commonly used alternatives

Different experimental designs may be more or less economical in different situations. Some experimental designs use a lot of subjects compared to others, but require fewer experimental conditions. Others use fewer subjects but may require more experimental conditions. How economical an experimental design is in your situation depends on the cost associated with each experimental subject relative to the overhead associated with an experimental condition.

To make the argument that the design you have chosen is the most economical one, it can be compelling to make a head-to-head comparison of the alternatives you considered. This can be done succinctly by including a table that lists the design alternatives you considered and shows how many subjects and how many experimental conditions each would require. For an example, look at Table 2 in Collins et al. (2011). This table shows that for that study, the individual experiments approach would have required 3,072 subjects, whereas the factorial design we chose required only 512 subjects. On the other hand, individual experiments would require 12 experimental conditions, as compared to 32 required by the fractional factorial design we implemented in this study.

If you know or can reliably estimate the costs associated with subjects and experimental conditions, you can express this comparison in terms of money. Table 2 in Collins et al. (2011) shows that in that study we saved the expense associated with 2,560 subjects by conducting a factorial experiment rather than individual experiments. If, for example, we had planned to pay each subject $50, this would have represented a savings of $128,000 in subject costs. However, this would have been offset by the costs associated with the increased number of experimental conditions. For example, suppose the overhead associated with each condition was $2,000. Then the experimental condition overhead would be $40,000 more in the factorial experiment than in the individual experiments approach. The net savings with a factorial experiment compared to the individual experiments approach would be $128,000-$40,000=$88,000.

Collins, Dziak, and Li (2009) explain how to compute figures like the ones in Table 2 in Collins et al. (2011). I suggest reading this article carefully. Although the numbers are easy to calculate (you may even be able to do the computations in your head), the rationale behind them is both subtle and complex. It is important to understand this rationale to ensure that you are applying the ideas correctly in your situation. Also, note that the expressions in Collins et al. apply only to designs with two levels per factor.

We’ve developed a SAS macro to help you to select the most economical design from among four alternatives that are potentially useful in intervention science (individual experiments, single factor experiments such as comparative treatment designs, factorial experiments, and fractional factorial experiments).

Planning a factorial experiment when there is a subgroup structure in the data (subjects are clustered within schools, clinics, etc.) is discussed in Dziak, Nahum-Shani, and Collins (2012).

References

Collins, L. M., Baker, T. B., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., Schlam, T. R., Cook, J. W., & Fiore, M. C. (2011). The multiphase optimization strategy for engineering effective tobacco use interventions. Annals of Behavioral Medicine, 41, 208-226. PMCID: PMC3053423

Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202-224. PMCID: PMC2796056

Dziak, J. J., Nahum-Shani, I., & Collins, L. M. (2012). Multilevel factorial experiments for developing behavioral interventions: Power, sample size, and resource considerations. Psychological Methods. Advance online publication. doi: 10.1037/a0026972 PMCID: PMC3351535

 

Justify the use of a fractional factorial experiment

If you are planning to use any type of factorial experiment, it has to be appropriate to address the research questions at hand. The first step, then, is to justify the use of a factorial experiment. Once that is accomplished, the next step is to justify the use of a fractional factorial rather than a complete factorial.

The best justification for the use of a fractional factorial design rather than a complete factorial design is economy. Remember that complete and fractional factorial designs are powered exactly the same, so there will be no savings in terms of subjects. In other words, any fractional factorial design requires the same number of subjects as the corresponding complete factorial design to maintain the same level of power. Fractional factorial designs involve fewer experimental conditions, so you should be able to argue that having fewer experimental conditions will result in savings of money, personnel, equipment, materials, logistical or management difficulties, etc. The more specific you can be about this in a grant proposal, the better.

Even when a complete factorial involves a lot of experimental conditions, a fractional factorial may not be more economical. For example, if you are conducting an internet-based experiment, it may not be that much more difficult to program 32 conditions than to program 16 conditions. (To repeat: Although this may be counterintuitive, the 32-condition experiment will require no more subjects than the 16-condition experiment.)

When writing a proposal it is a good idea to be explicit about the trade-offs associated with using a fractional factorial design, namely aliasing of effects (for an explanation of aliasing, see Collins et al., 2009). Whenever you remove conditions from a complete factorial design aliasing occurs. When you select a fractional factorial design, you are choosing aliasing in a strategic fashion, and saying that the aliasing is an acceptable trade-off for increased economy.

It might help to remind reviewers that from one perspective, designs such as the familiar comparative treatment, constructive treatment, and dismantling designs also involve aliasing. These designs have properties that are not nearly as attractive as a good fractional factorial; see Collins, Dziak, and Li (2009).

Reference

Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202-224. PMCID: PMC2796056

 

Explain the rationale behind your choice of a fractional factorial design

There are a lot of different fractional factorial designs with different aliasing structures. (If you do not know what the term aliasing means, read Collins, Dziak, and Li, 2009.) You can use software like PROC FACTEX in SAS to select a design. We offer a mini-tutorial about doing this in Collins et al. (2009).

You might want to consider a 25-1 design. I really like this design and have been recommending it lately to intervention scientists. It enables examination of 5 factors, but requires only 16 experimental conditions (a complete factorial design would require 32 conditions, so this is a half fraction). The design has the following important characteristics:

  • each effect is aliased with ONE other effect (because the design is a half fraction); and
  • It is Resolution V; in this case this means that each main effect is aliased with one four-way interaction, and each two-way interaction is aliased with one three-way (or higher-order) interaction.

It seems to me that this aliasing is pretty easy to justify. How many people would argue that four-way interactions are likely to be large and important? If some of the two-way interactions are scientifically important, make sure that they are aliased with three-ways that are likely to be negligible in size.

Reference

Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202-224. PMCID: PMC2796056

 

Convince biostatisticians who may be reviewers that a factorial or fractional factorial experiment is a good idea

Many biostatisticians are steeped in the RCT tradition, but they are usually quick to see that factorial experiments have advantages in some situations when the argument is presented to them. These articles are likely to appeal to “card-carrying” biostatisticians:

Chakraborty, B., Collins, L. M., Strecher, V., and Murphy, S. A. (2009). Developing multicomponent interventions using fractional factorial designs. Statistics in Medicine, 28, 2687-2708.

Nair, V., Strecher, V., Fagerlin, A., Ubel, P., Resnicow, K., Murphy, A., Little, R., Chakraborty, B., & Zhang, A. (2008). Screening experiments and the use of fractional factorial designs in behavioral intervention research. American Journal of Public Health, 98, 1354-1359.

 

Convince reviewers that your factorial experiment has sufficient power, keeping in mind that some reviewers believe that factorial experiments can never be sufficiently powered

Reviewers who are simply unfamiliar with factorial experiments often come into a review with an open mind. They are likely to be convinced by a clear argument about the appropriateness and economy of a factorial experiment. However, some reviewers ardently believe “facts” about factorial experiments that are wrong. In particular, some reviewers look at a 25 factorial experiment and see a 32-condition RCT. Because it would be nearly impossible to power a 32-condition RCT, they may conclude that it is impossible to power a 25 factorial experiment. Fortunately, as behavioral scientists re-learn the logic behind factorial experiments there are likely to be fewer and fewer reviewers who make this incorrect conclusion.

Recently I saw a case in which it appeared from the summary statement that one reviewer had immediately assumed it would be impossible to power the factorial experiment proposed, but in the discussion the other reviewers explained why there was adequate power. The reviewer’s mind was changed, and this change of mind was reflected in the priority score. So, there is hope!

If you can spare the room in the proposal (a big “if,” I know), it may be a good idea to include a brief tutorial about factorial experiments. Here are the main points I’d suggest covering.

  • Explain that the factors are crossed, and include a table showing the experimental conditions in the design you are proposing to use (it is helpful to number the conditions). It may seem silly to include a table of experimental conditions, but I have found that many people do not know how to get from a list of factors to a set of experimental conditions. Such people will not be able to visualize your design without the help that a table will provide.
  • Based on the table of experimental conditions, illustrate how to compute a main effect estimate (e.g. “The main effect of Resistance Training is the mean of Experimental Conditions 1,3,5, and 7 minus the mean of Experimental Conditions 2,4,6, and 8”).
  • Point out that the purpose is NOT to compare the means of these conditions to each other directly, as it would be in an RCT. Instead, the experimental conditions will be combined in various ways to estimate main effects and interactions in the ANOVA. Importantly, because of the way conditions are combined, each main effect and interaction is based on the entire set of subjects. This is why factorial experiments make such economical use of subjects (you can cite Collins, Dziak, & Li, 2009).

Here is an example paragraph you can use as a starting point. This is based on a 16-condition factorial experiment, either a complete 24 factorial or a fractional factorial with more than four factors. I’m assuming you have included Table X showing all the experimental conditions in the design. Suppose two of the factors are NORM and RESIST, and it has been determined that there will be sufficient power with a total N=800:

We ask the reader to note that the factorial design in Table X should not be considered a 16-arm trial in which each condition is compared in turn to a control condition. Our interest is in tests of standard ANOVA main effects and interaction effects. These tests involve comparison of means computed across aggregates of experimental conditions. For example, the main effect of the NORM component will be tested by comparing the mean of the outcome variable for the 400 subjects who receive NORM (i.e., those in Conditions 1-8 in Table X) versus the mean of the outcome variable for the 400 subjects who do not receive NORM (i.e., those in Conditions 9-16). Similarly, the main effect of RESIST will be tested by comparing the mean of the outcome variable for the 400 subjects who receive RESIST (i.e., Conditions 2,3,5,8,9,12,14,15) versus the mean of the outcome variable for the 400 subjects who do not receive RESIST (i.e., Conditions 1,4,6,7,10,11,13,16). Analysis of data from a 16-arm RCT would require comparison of individual experimental conditions, and therefore would be grossly underpowered. However, because we are conducting a multi-factorial experiment and not an RCT, each effect estimate will involve all 16 of the conditions in Table X, thereby maintaining the power associated with all 800 subjects.

Read a brief, informal webpage that explains factorial experiments for those trained primarily in RCTs. For a brief article on this topic, see Collins, Dziak, Kugler, and Trail (2014).

Reference

Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202-224. PMCID: PMC2796056

Collins, L. M., Dziak, J. J., Kugler, K. C., & Trail, J. B. (2014). Factorial experiments: Efficient tools for evaluation of intervention components. American Journal of Preventive Medicine, 47, 498-504.

 

Cite the evidence that implementing a large factorial experiment in a field setting can be manageable

With careful planning and organization, thorough training of staff, good management, and creative use of technology, it is possible to implement large factorial experiments in field settings successfully. I recommend relying primarily on professional implementation staff rather than graduate students. Listed below are published articles that describe factorial experiments in varying stages of completion. As I learn of others I will add them. If you know of other implementations of factorial experiments WITH AT LEAST THREE FACTORS in the behavioral intervention area, please email the citations to Linda Collins for consideration for inclusion here.

Baker, T. B., Collins, L. M., Mermelstein, R., Piper, M. E., Schlam, T. R., Cook, J. W., et al. (2016). Enhancing the effectiveness of smoking treatment research: conceptual bases and progress.Addiction, 111(1)., 107-16. doi:10.1111/add.13154

Caldwell, L. L., Smith, E. A., Collins, L. M., Graham, J. W., Lai, M., Wegner, L., Vergnani, T., Matthews, C., & Jacobs, J. (2012). Translational research in South Africa: Evaluating implementation quality using a factorial design. Child and Youth Care Forum, 41, 119-136. PMCID: PMC3375728

Collins, L. M., Baker, T. B., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., Schlam, T. R., Cook, J. W., & Fiore, M. C. (2011). The multiphase optimization strategy for engineering effective tobacco use interventions. Annals of Behavioral Medicine, 41, 208-226.

Cook, J. W., Collins, L. M., Fiore, M. C., Smith, S. S., Fraser, D., Bolt, D. M., et al. (2016). Comparative effectiveness of motivation phase intervention components for use with smokers unwilling to quit: a factorial screening experiment. Addiction, 111(1)., 117-28. doi:10.1111/add.13161

Piper, M. E., Fiore, M. C., Smith, S. S., Fraser, D., Bolt, D. M., Collins, L. M., et al. (2016). Identifying effective intervention components for smoking cessation: a factorial screening experiment. Addiction, 111(1)., 129-41. doi:10.1111/add.13162

Schlam, T.R., Fiore, M.C., Smith, S.S., Fraser, S., Bolt, D.M., Collins, L.M., Mermelstein, R., Piper, M.E., Cook, J.W., Jorenby, D.E., Loh, W.-Y., & Baker, T.B. (2016). Comparative effectiveness of intervention components for producing long-term abstinence from smoking: A factorial screening experiment. Addiction, 111, 142-155. PMCID: PMC4692280 doi: 10.1111/add.13153

Strecher, V. J., McClure, J. B., Alexander, G. W., Chakraborty, B., Nair, V. N., Konkel, J. M., Greene, S. M., Collins, L. M., Carlier, C. C., Wiese, C. J., Little, R. J., Pomerleau, C. S., Pomerleau, O. F. (2008). Web-based smoking cessation programs: Results of a randomized trial. American Journal of Preventive Medicine, 34,373-381.