Suppose I have a smoking cessation intervention. For the purposes of demonstration, let us say that my intervention consists of consuming a Vienna Finger every time a smoker trying to quit has a desire for a cigarette. I assemble a group of 100 people who want to quit and enter them into my Vienna Finger-assisted smoking cessation program.
After 6 months, I want to assess the percentage of participants who have successfully quit smoking. So I send out a survey to the 100 participants asking them if they are smoking at the present time.
Let's say I receive 20 surveys back, and 10 of those 20 respondents indicate that they have quit smoking.
I then have two basic choices for reporting the quit rate for my intervention:
First, I could base the quit rate only on those who have responded to the survey. There were 20 respondents and 10 indicated that they quit, so I could boast that the quit rate for my intervention is 50%. This is called a responder analysis or a responder quit rate.
Second, I could base the quit rate on all those who participated in the program. There were 100 participants, but I can only confirm that 10 of them quit smoking. Eighty respondents were lost to follow-up in the sense that they did not complete the evaluation survey. Chances are that a high proportion of the non-responders failed to quit smoking. One way to handle this is to assume that every non-responder failed to quit smoking. This is called an "intention-to-treat" analysis. The intention-to-treat quit rate would be 10 confirmed quitters out of 100 participants, or 10%.
You can see that with a low response rate, there is a vast difference between the responder quit rate and the intention-to-treat quit rate. What you need to recognize is that the true quit rate is somewhere in between the two quit rates. But most likely, it is much closer to the intention-to-treat quit rate. Why? Because research has demonstrated that non-responders are much, much more likely to be continuing or relapsing smokers.
The responder quit rate assumes that the rate of quitting among non-responders is the same as that among responders. This is an untenable assumption because non-response is almost certainly differential with respect to smoking status. In other words, those who are successful quitting are probably more excited about filling out the survey to tell you how successful the program was. Those for whom the program failed are much less likely to be excited and motivated to take the time to complete the survey.
Any company which only provides its responder quit rate is misrepresenting the true quit rate, unless the response rate to the survey was extremely high. In the case of Free & Clear, it appears that the survey response rate was not high enough to allow the company to use the responder quit rate as a valid indication of the effectiveness of the program.