Intervention Component Selection Experiments in the Optimization Phase of MOST

What experimental designs are used in MOST?

There are many different approaches to experimentation that can be used in the optimization phase.  Factorial experiments, fractional factorial experiments, sequential multiple-assignment randomized trials (SMARTs), micro-randomized trials, system identification, or any other suitable approach can be used.  MOST does not require any particular approach to experimentation in the optimization phase, only that the approach selected is highly efficient, in other words, is the best one according to the resource management principle.  The choice of approach depends on the type of intervention that is to be optimized, the exact empirical information that is needed, and the resources that are available to conduct the experimentation.

What are the component selection experiments/screening experiments that are part of the optimization phase of MOST?

In the preparation phase of MOST the investigator develops a set of intervention components that  are candidates for inclusion in the intervention. As part of the process of developing an optimized intervention, the investigator has to make decisions about these components.  In some cases the investigator needs to decide whether a particular component is to be included at all (e.g., whether to include a parent component in a school-based intervention). In other cases the investigator has already decided to include the component, but needs to decide which level of the component is most appropriate (e.g., whether a brief version of motivational interviewing is sufficient, or a more extensive session is needed).

In the optimization phase the investigator gathers empirical information that will form the basis for making these decisions. This information is obtained via a component selection experiment. Component selection experiments are highly efficient, sufficiently powered, well-controlled, practical, and relatively quick (although, how quick they can be varies considerably by field; see How can a complete cycle of MOST (that is, all three phases) be conducted within the five-year limit on NIH grants, and it is necessary to do so?). They are aimed at making the best use of research resources (money, time, experimental subjects, etc.) to gain the most useful information.

In early articles on MOST we used the term “screening experiment” because it is used widely in engineering. However, it caused confusion for many intervention scientists, who are accustomed to using the term screening to refer to criteria for subject inclusion or exclusion in a study, or criteria for referral to treatment. To avoid this confusion we have recently begun using the term “component selection experiment” instead. Both terms are still used sometimes.

Do I have to conduct a factorial experiment as the component selection experiment?

Definitely not. According to the resource management principle, you should use the most efficient experimental design that enables you to address the research questions you have identified as most important. In writing about MOST we have emphasized factorial designs because they tend to be efficient, and because they have largely been overlooked by intervention science. However, it is impossible to make a blanket recommendation about experimental design because no single design approach is the best one for every situation.

What are the most important considerations when choosing an experimental design for MOST?

First, it is important to select an experimental design that addresses the most important research questions at hand. Different designs often address subtly different research questions.

Second, it is important to select an experimental design that is efficient, that is, makes the best use of available resources to address the key scientific questions. Different kinds of resources may be available in different situations. In some areas of research, subjects are expensive or hard to get. In others, the overhead costs associated with each experimental condition may be high. Different experimental designs demand different kinds of resources, so a design that is very economical in one setting may be expensive in another.

Where do pilot studies fit into MOST?

The answer to this question depends on your definition of the term “pilot study.” Different people, even people from the same field, often have different definitions of this term.

Those of us working in MOST prefer not to refer to component selection experiments as pilot studies, to avoid ambiguity about the definition of a pilot study that might lead to misunderstandings about the role of these experiments in MOST. It is critical for MOST that the component selection experiments are serious, fully powered, carefully conducted, and tightly controlled experiments.

Many people subscribe to Vogt’s (1993) definition of the term pilot study:

“A preliminary test or study to try out procedures and discover problems before the main study begins. This enables researchers to make last-minute corrections and adjustments. It is a research project’s ‘dress rehearsal.’” (p. 172)

This definition suggests that a pilot study is not as carefully controlled as the “real thing;” a pilot study might not even involve random assignment. It also suggests that statistical power is not much of a consideration in a pilot study, because there will be no formal hypothesis testing. Conducting this kind of pilot study to try out individual intervention components before conducting any kind of experiment is a good idea. This is something most careful intervention scientists have always done before conducting an RCT, and it should be done as part of the preparation phase in MOST, before conducting the component selection experiment.

We wish to make the point that informal pilot studies conducted in advance of an RCT, valuable though they are, do not constitute MOST.

Reference

Vogt, W. P. (1993). Dictionary of statistics and methodology: A nontechnical guide for the social sciences. Newbury Park: Sage.

Does the MOST framework include use of the randomized controlled trial (RCT)?

Definitely. Generally an RCT is used to complete the evaluation phase of MOST.

You may use figure 1 in presentations, online, or in publications free of charge.

Is there such a thing as a “MOST design”?

It depends on what it meant by the term “design.”  If “design” is referring to an overall approach to or framework for conducting research, then the answer is yes.  However, if “design” is referring to a particular experiment design, then the answer is no. (See “Do I have to conduct a factorial experiment as the component selection experiment?”) Occasionally people call a factorial design a MOST design. PLEASE DO NOT DO THIS. Factorial experiments are used in many contexts besides MOST. We definitely did not invent factorial experiments (they were invented by R.A. Fisher in the 1920’s)!

What should I use as an outcome variable for the component selection experiment?

The best outcome variable is a direct measure of the public health or educational outcome. For example, if you are building a drug abuse prevention intervention, it is best to use drug use behavior as an outcome variable.

However, often using a measure of the ultimate public health outcome is not practical for the component selection experiments, because it would take too long. For example, in drug abuse prevention, the intervention is often delivered in 5th or 6th grade, and the drug use outcome may not be measurable until 9th or 10th grade. Waiting four years to assess the outcome is often reasonable for an RCT, but it is too long to wait for the outcome of a component selection experiment.

This situation illustrates the critical importance that the theoretical model plays in MOST. Nearly every behavioral intervention is hypothesized to operate by affecting mediating variables which in turn affect the outcome. In drug abuse prevention, a program might be designed to correct beliefs about how normative drug use is, and to improve skills for resisting offers to use drugs. These mediating variables can be used as short-term outcome variables for the component selection experiment. The logic here is that if the intervention is being designed to have an effect on these mediators, it should demonstrate this effect empirically; if the intervention is not having an effect on the hypothesized mediators, it will not have an effect on the outcome.

A good theoretical model will specify which components are expected to have an effect on which mediating variables. Thus the model can be used to derive a priori hypotheses about which intervention components are expected to affect which mediators, and which are expected not to affect which mediators. This in turn can inform the selection of short-term outcomes for the component selection experiment.  (See “What are the features of the theoretical model that is needed for the Preparation phase of MOST?” and “Do I really need a theoretical model to use the MOST framework?“)

Of course, there is some risk associated with decisions such as these, but taking calculated risks to move science forward faster is a part of MOST; see “What is the resource management principle?”.

Return to MOST home page.

Read practical questions about using MOST.