The nutrition space can be extremely noisy. At Needed, we are committed to highlighting nutrition fundamentals over fads, and we want to empower you to do the same.
Today, we are sharing the second of three articles on Finding, Evaluating, and Understanding Scientific Research. In this post, we delve a little deeper into how to determine if a paper presents legitimate and clinically-sound research. While that may seem overwhelming, there are several easy-to-identify factors to look for when evaluating the validity of a scientific paper.
Reputation of Journal:
If the paper has been published in a highly reputable journal, you can generally assume it contains legitimate and sound research. This is because the studies within these journals are subject to rigorous peer review and must meet the editors’ high standards before being considered for publication. There are many reputable, highly-ranked journals in both conventional and natural medicine that can be evaluated via the following:
- Journal Impact Factor ratings is the most reliable way to vet a journal, but access to this info requires a subscription.
- You can access any number of websites that rank scientific journals, such as Scientific Journal Rankings here.
Reputation and Relevance of Paper:
Two other important factors to consider when evaluating a paper include the number of times the study has been cited in other research articles and the date of publication. These two factors speak to the reputation and the relevance of the paper, respectively.
- Check to see how many times a paper has been cited in other research. There is no exact number that offers credibility to a study, but the more times cited, the more likely the research was well-received and respected by other researchers in the field. It is important to note that number of citations also correlate with relevance of the topic. For this reason, a well-researched and legitimate article may have few citations if the topic being studied is not of interest or being studied by other researchers.
- Look at how long ago the research was published and if the paper is still being cited
- If the study is >10 years old and no longer being cited, it indicates that more current research has been published that negates or surpasses earlier information. Thus, older studies are typically less relevant, as new information tends to build on previous understandings.
Sound Methodologies: Various types of study methodologies exist depending on study design. For the purpose of this series, we are focusing on experimental research, as opposed to observational research. Experimental studies are considered to be more clinically-sound than observational studies because they assess cause and effect, rather than simply identifying correlations. Of all experimental designs, randomized, double-blind, placebo-controlled (RDBPC) trials are the gold standard for clinical trials. However, RDBPC’s are the most expensive type of study to conduct and usually the last to be done after other phases of clinical trials have been completed. When examining an experimental research study for legitimacy, the following should be considered:
- Materials: The study materials being tested (nutrient, botanical, vitamin) should be from a reputable supplier, and the selected placebo controls should be well blinded. In other words, the placebo control material should look, taste, and smell identical to the test material so subjects can’t tell them apart.
- Controlled Variables: One of the most important hallmarks of sound research is the control for outside variables. For this reason, there is a hierarchy of research based on the specificity of the item and/or outcome being studied. In regards to nutrition, isolated nutrient-based research is at the top, followed by food-based research, and finishing with cultural diet-based research. This is because nutrient-based research has the least number of erroneous variables that need to be controlled. In the same way, research studying acute onset disease are considered to be more valid than studies looking at chronic disease, as the cause of chronic disease is multifactorial and thus more difficult to study. For this reason, a study that evaluates a highly specified item or outcome is considered to be a better clinical trial.
- Study Participants: Depending on the type of clinical trial, method of participant selection and population size will vary, but sound clinical trials typically meet the following guidelines:
- Number of Participants: Study populations can range from 1 individual to thousands of participants. The higher the population, the more statistically significant the results. In general, reputable studies have at least 30 participants.
- Randomized: If comparing the results of two separate groups (i.e. experimental and control), it is important that individuals or objects were assigned to a group in random manner. This helps eliminate research bias.
- Study duration: The length of the study can also vary depending on the type of trial being conducted. Pilot trials may only involve a single dose given on one day or over a short period of time; whereas, some RDBPC trials can last a year or more. Study duration will also depend on the outcome being studied. When evaluating safety, the study should be carried out for a minimum of 30 days. When evaluating pharmacokinetics, a single-dose or short term (1-2 weeks) trial may be sound. Generally, longer studies are considered to be more methodologically sound.
- Replication: The method by which data is collected should be clear and replicable. This means that other researchers should be able to perform the study and get the same results after reading the methods section.
Outcomes/Results are Measurable and Appropriate for Study Design
The measured outcomes (aka results) should be reasonable and based on the hypothesis and parameters being studied. This means that the method of collecting data must provide an accurate way of assessing a quantifiable difference from pre- to post-intervention. Several types of outcomes exist, all of which can have significant value:
- Some studies have only primary outcomes (i.e. study material was safe and well tolerated).
- Others have secondary outcomes (i.e. study material was safe and improved nutrient status in the general population).
- Some trials even measure tertiary outcomes in perhaps a sub-group within the large group (i.e. study material was safe, improved nutrient status, and reduced blood pressure in a subgroup of menopausal women).
Other important factors to look in regards to the reporting of results are as follows:
- Results presented should not make assumptions based on previous research.
- Tables and graphics of results should be clear and accurate.
- All graphics (charts, graphs, images) should have a title and easily identifiable data points and thresholds/baselines.
Valid Statistical Analysis - Understanding P Value
Statistical analysis of data is a vital key to determining if a study was adequately “powered” to yield statistically significant results.
- Look for the P value in the Results section of a paper. This important number indicates whether the results are statistically significant or not.
- A P value ≤ 0.05 indicates statistically significant results were achieved and the research hypothesis is likely valid.
- A P value of ≥ 0.05 indicates results of test group were not statistically significant as compared to the results group, and the proposed hypothesis is likely rejected.
Conclusions Based on Results Should Illustrate and Recap the Findings.
Conclusions should not make unreasonable claims or inferences, but recap the studied material’s effect on desired outcomes, with an explanation offered for the impact or lack of impact of each study outcome.
A List of Properly Formatted References.
A properly formatted reference list gives credibility and legitimacy to the paper, acknowledges the work on the topic by other researchers, and helps to avoid plagiarism. References provide the reader sources to check the legitimacy of the paper. A properly formatted reference will include:
- Author name(s)
- Title of Study
- Publishing Journal
- Unique Journal Citation including year, journal issue, volume, and page numbers
