We love research at WorkGuru – we strongly believe that online programmes like ours should have the same evidence base as the medication that your doctor prescribes. But what do you need to look out for to ensure that the research you use to guide your daily lives or inform your purchasing decisions is good quality research?
If someone is quoting research as part of their evidence for why you should buy their programme or service, ask for a reference to the research so that you can look at it yourself. A lot of studies are now published as open access, which means that you don’t have to pay to see it; it is freely available for anyone to look at.
When looking at research ask yourself the following eight questions:
- What is the design of the study?
- Was the study done on humans?
- Do they have a control group?
- Are there enough participants in the study and whom do the participants represent?
- How is the data being analysed?
- Who funded and conducted the study?
- Was the study peer reviewed?
- Are the findings of the study being accurately reported?
The gold standard of medical research is the randomised control trial (RCT), this is where participants are randomly assigned to groups, but it isn’t always possible or ethical to run RCTs in public health or in psychology. If you were researching into smoking for example, it wouldn’t be ethical to randomly assign a group as smokers. Not all studies need to be RCTs but the best ones will have a control group. We are currently running an RCT looking at how people engage with WorkGuru.
Another question to ask is whether the study involves humans. This may sound strange, but there are examples where studies have been conducted using rats or zebrafish and the reporting of the study in the press have neglected to mention that. One example is the often-quoted study exalting the benefits of champagne drinking on memory, rarely mentioned in the press is that the study was conducted using aged rodents.
The number of people that participated in the study is also important. Studies with just a few participants are unlikely to be statistically significant (there is a possibility that they could have got the same results by chance alone), and they are unlikely to be representative of a broader population.
Next, see if there is any information on how the study data was analysed. One thing you can look for is whether the data for all the participants was analysed (often called intention to treat or ITT) or just for the people who completed the study (per protocol). If you only analyse the data from participants who fully comply with the study, you may get different results than if you include the data for everyone – even those who dropped out or didn’t complete all the questionnaires. It is also worth looking at who conducted or funded the study. Research espousing the health benefits of chocolate isn’t necessarily inaccurate if it is funded or conducted by a chocolate manufacturer, but having that information may influence how credible you think the research is. What steps have the researchers taken to ensure that the research isn’t biased?
Where the study is published is another factor that can tell you about the quality of the study. The majority of academic journals are peer reviewed. Peer review is where academics that are expert in the field read and comment on the paper before the journal decides whether to publish it. Peer review isn’t without its problems, but it is a fairly good indictor of quality.
And finally, is the study accurately reported. Does the academic paper actually say what it is being quoted as saying?
It is important that the evidence base guides us, but it is also important to recognise that not all research is equal, and that some research is more likely to be accurate than other research. Really good quality research can be replicated and get the same results. One of our recent blogs talks about the fickleness of research and gives examples of much loved and influential research that has failed to replicate.
0 Be the first to recommend this post