“Undefined terms in systematic reviews make them unscientific and unreliable”

• Claim: Undefined terms in systematic reviews make them unscientific and unreliable
• Verdict: MISLEADING
• Evidence Level: L2 — Academic guidelines and methodological articles from peer-reviewed sources
• Key Anomaly: The claim conflates methodological rigor with definitional completeness. Systematic reviews can be scientifically sound even with undefined elements, provided this uncertainty is explicitly acknowledged and methodology remains rigorous
• 30-Second Check: Authoritative sources (UCL, BMJ Paediatrics Open) confirm that systematic reviews should either specify definitions at the outset OR clearly state which elements remain undefined — both approaches are methodologically acceptable (S006, S003)

Steelman — What Proponents Claim

The strongest version of this claim rests on a fundamental principle of scientific method: reproducibility and precision require clear definitions. Proponents might argue that systematic reviews, which claim to synthesize scientific evidence, must operate with strictly defined terms, otherwise:

• Inclusion and exclusion criteria become arbitrary
• Different researchers cannot reproduce the literature search
• Review conclusions lose generalizability due to conceptual ambiguity
• Readers cannot assess applicability of results to their context

This position finds partial support in methodological literature. Guidelines for conducting systematic reviews do emphasize the importance of explicit inclusion criteria and clearly described study types (S005). University library guides highlight the necessity of comprehensive literature searches and explicit selection criteria as distinguishing features of systematic reviews compared to narrative reviews (S005, S008).

Moreover, philosophical literature acknowledges the problematic nature of undefined terms in logical argumentation. The Stanford Encyclopedia of Philosophy notes that while some basic concepts such as "relevance" and "sufficiency" remain intuitive and undefined even in formal logic, this creates methodological difficulties (S013).

What the Evidence Actually Shows

Empirical data and methodological guidelines paint a more nuanced picture that refutes the categorical nature of the original claim.

Uncertainty as Recognized Methodological Reality

University College London's guidance on formulating research questions for systematic reviews contains a critically important clarification: "Any research question has ideological and theoretical assumptions around the meanings and processes it is focused on. A systematic review should either specify definitions and boundaries around these elements at the outset, or be clear about which elements are undefined" (S006).

This guidance, updated in January 2026, explicitly recognizes the legitimacy of two approaches: complete definition or explicit acknowledgment of uncertainty. The key requirement is transparency, not absolute certainty of all terms.

Empirical Examples of Scientifically Sound Reviews with Undefined Concepts

The systematic review by Kloos and colleagues (2023), published in Medical Teacher and cited 15 times, examines "neglect as an undefined and overlooked aspect of medical student mistreatment" (S004). The title itself acknowledges that the central concept of the study — "neglect" — is undefined. Nevertheless, this is a peer-reviewed systematic review published in a reputable medical education journal, demonstrating that the scientific community accepts systematic reviews that explicitly work with undefined concepts.

This example illustrates an important methodological principle: systematic reviews can serve as tools for investigating and clarifying previously undefined concepts, not only for synthesizing data on already clearly defined questions.

Critique of False Dichotomy Between Review Types

Gordon's (2025) article in BMJ Paediatrics Open provides critical analysis of a common misconception: "Any undefined question will, by default, encompass a wide scope—but that alone does not justify a scoping review. More concerningly, many submissions under the scoping label discard essential elements of rigour, which are vital for producing findings that are reliable and generalisable" (S003).

This observation inverts the logic of the original claim. The problem is not that undefined terms make reviews unscientific, but that researchers sometimes use uncertainty as a pretext for reducing methodological rigor. Gordon emphasizes that wide scope or question uncertainty does not exempt from the necessity of maintaining "essential elements of rigour" (S003).

Methodological Standards Independent of Definitional Completeness

Comparative analysis of review types shows that systematic reviews differ from narrative reviews not in definitional completeness, but in methodological characteristics (S005):

• Comprehensive literature search: designed to locate all relevant studies
• Explicit inclusion criteria: clear description of study types to be included
• Systematic data extraction: structured process for extracting information
• Quality assessment: formal evaluation of methodological quality of included studies

None of these criteria require that all terms be fully defined before beginning the review. They require transparency, reproducibility, and systematicity of process.

Conflicts and Uncertainties in the Evidence

Limitations of Extracted Data

A substantial limitation of available sources must be acknowledged: most extracted materials consist of fragmentary texts from library guides, containing predominantly navigational elements and incomplete excerpts. Only four sources (S003, S004, S006, S010) contain substantive content suitable for methodological analysis.

This limitation means that this analysis rests on a relatively narrow evidence base, although the quality of available sources (peer-reviewed journals, prestigious university guidelines) remains high.

Tension Between Ideal and Practice

There exists an obvious tension between the methodological ideal of complete certainty and the practical reality of the research process. The systematic review and meta-analysis by Bagde and colleagues (2023) on ChatGPT, cited 73 times, notes that "some of the studies have undefined locations or sample sizes" (S010).

This observation points to a practical problem: even in contemporary systematic reviews published in authoritative journals, researchers encounter uncertainty in primary data. The question is not whether uncertainty can be completely avoided (often it cannot), but how to handle it in a methodologically correct manner.

Philosophical Foundations of Uncertainty

The Stanford Encyclopedia of Philosophy acknowledges a fundamental problem: some basic concepts remain "intuitive, undefined concepts" even in formal logic (S013). This philosophical observation has important implications for systematic review methodology: if even in mathematical logic there exist inevitably undefined terms, requiring absolute certainty in empirical research may be unrealistic.

Moreover, the source on circular definitions notes that dictionary structure proves "the existence of undefined terms, terms whose meanings we understand without formal definition" (S015). This indicates that some degree of uncertainty is an inevitable characteristic of language and conceptual systems.

Interpretation Risks and Practical Implications

Danger of False Dichotomy

The original claim creates a false dichotomy: either all terms are fully defined (and the review is scientific), or undefined terms are present (and the review is unscientific). Evidence shows that reality is much more complex. The scientific nature of a systematic review is determined by:

• Transparency of methodology
• Reproducibility of search and selection process
• Systematicity of data extraction and analysis
• Explicit acknowledgment of limitations and uncertainties
• Rigor of quality assessment of included studies

Definitional completeness is one factor, but not the only or always decisive criterion of scientific validity.

Risk of Research Paralysis

If the claim is taken literally, it could lead to paralysis of research in new or interdisciplinary fields where conceptual frameworks are still forming. The example of the systematic review on "neglect" in medical education (S004) shows that systematic reviews can serve as tools for developing conceptual clarity, not only for synthesizing already clearly defined knowledge.

Substitution of Methodological Rigor with Conceptual Certainty

The most serious risk is that focus on term certainty may distract attention from more fundamental methodological problems. Gordon (2025) warns that researchers sometimes use question uncertainty as justification for reducing methodological rigor (S003). The real problem is not uncertainty itself, but using uncertainty as a pretext for abandoning "essential elements of rigour."

Contextual Dependence of Certainty Requirements

Different research fields and types of questions require different levels of conceptual certainty. Clinical questions about specific medical interventions may require stricter definitions than research questions about social phenomena or qualitative aspects of experience. A universal requirement for absolute certainty ignores this contextual variability.

Conclusions for Practice

For researchers conducting systematic reviews, evidence points to the following practical recommendations:

1. Transparency matters more than absolute certainty: Explicitly state which elements are defined and which remain undefined (S006)
2. Uncertainty does not justify reduced rigor: Wide scope or undefined question does not exempt from the necessity of comprehensive search, explicit criteria, and systematic assessment (S003)
3. Use systematic reviews to clarify concepts: Reviews can serve as tools for investigating and defining previously undefined phenomena (S004)
4. Follow established standards: Cochrane guidelines, PRISMA, and institutional protocols ensure methodological rigor independent of degree of conceptual certainty (S007, S002)

For consumers of systematic reviews, it is critically important to assess methodological rigor of the process, not only completeness of definitions. A well-conducted systematic review with explicitly acknowledged undefined elements is more reliable than a methodologically weak review with formally defined but arbitrarily chosen terms.