Evaluate scientific claims and evidence quality. Use for assessing experimental design validity, identifying biases and confounders, applying evidence grading frameworks (GRADE, Cochrane Risk of Bias), or teaching critical analysis. Best for understanding evidence quality,
Installs just this skill. Get the whole plugin for auto-invocation.
⚡ How it fires
How this skill gets triggered: by you, by Claude, or both.
Fires itselfClaude auto-loads it when your prompt matches the work.
You can call itInvoke it directly when you want it.
Slash command/scientific-critical-thinking
👁️ Context preview
The summary Claude sees to decide when to auto-load this skill.
Evaluate scientific claims and evidence quality. Use for assessing experimental design validity, identifying biases and confounders, applying evidence grading frameworks (GRADE, Cochrane Risk of Bias), or teaching critical analysis. Best for understanding evidence quality,
📊 Stats
Stars31,549
Forks3,146
LanguagePython
LicenseMIT
📦 Ships with scientific-agent-skills
</> SKILL.md
scientific-critical-thinking.SKILL.md
---name: scientific-critical-thinking
description: Evaluate scientific claims and evidence quality. Use for assessing experimental design validity, identifying biases and confounders, applying evidence grading frameworks (GRADE, Cochrane Risk of Bias), or teaching critical analysis. Best for understanding evidence quality, identifying flaws. For formal peer review writing use peer-review.
allowed-tools: Read Write Edit
license: MIT license
compatibility: Analytical guidance needs no network. Optional figures via the scientific-schematics skill require OPENROUTER_API_KEY and outbound API access to OpenRouter.
metadata: {"version": "1.1", "skill-author": "K-Dense Inc."}
---# Scientific Critical Thinking
## Overview
Critical thinking is a systematic process for evaluating scientific rigor. Assess methodology, experimental design, statistical validity, biases, confounding, and evidence quality using GRADE and Cochrane ROB frameworks. Apply this skill for critical analysis of scientific claims.
## When to Use This Skill
This skill should be used when:
- Evaluating research methodology and experimental design
- Assessing statistical validity and evidence quality
- Identifying biases and confounding in studies
- Reviewing scientific claims and conclusions
- Conducting systematic reviews or meta-analyses
- Applying GRADE or Cochrane risk of bias assessments
- Providing critical analysis of research papers
## Visual Aids (Optional)
Only add figures when the **user explicitly requests** a diagram (for example, a GRADE flowchart, bias decision tree, or evidence-quality framework).
**When figures help:**
- Critical thinking framework diagrams
- Bias identification decision trees
- Evidence quality assessment flowcharts
- GRADE or risk-of-bias evaluation frameworks
**How to create figures:**
- **Preferred:** Use the **scientific-schematics** skill for AI-generated diagrams from a natural-language description
- **Alternative:** Build figures in your usual tools (draw.io, PowerPoint, matplotlib, etc.)
From the `scientific-schematics` skill directory, with `OPENROUTER_API_KEY` set:
```bash
python scripts/generate_schematic.py "GRADE evidence assessment flowchart with downgrade and upgrade factors" -o figures/grade_flowchart.png --doc-type report
```
**Disclosure:** AI schematic generation sends your prompt to [OpenRouter](https://openrouter.ai/) (a third-party API). Do not include unpublished sensitive details unless that transmission is appropriate for your project.
---
## Core Capabilities
### 1. Methodology Critique
Evaluate research methodology for rigor, validity, and potential flaws.
**Apply when:**
- Reviewing research papers
- Assessing experimental designs
- Evaluating study protocols
- Planning new research
**Evaluation framework:**
1. **Study Design Assessment**
- Is the design appropriate for the research question?
- Can the design support causal claims being made?
- Are comparison groups appropriate and adequate?
- Consider whether experimental, quasi-experimental, or observational design is justified
2. **Validity Analysis**
- **Internal validity:** Can we trust the causal inference?
- Check randomization quality
- Evaluate confounding control
- Assess selection bias
- Review attrition/dropout patterns
- **External validity:** Do results generalize?
- Evaluate sample representativeness
- Consider ecological validity of setting
- Assess whether conditions match target application
- **Construct validity:** Do measures capture intended constructs?
- Review measurement validation
- Check operational definitions
- Assess whether measures are direct or proxy
- **Statistical conclusion validity:** Are statistical inferences sound?
- Verify adequate power/sample size
- Check assumption compliance
- Evaluate test appropriateness
3. **Control and Blinding**
- Was randomization properly implemented (sequence generation, allocation concealment)?
- Was blinding feasible and implemented (participants, providers, assessors)?
- Are control conditions appropriate (placebo, active control, no treatment)?
- Could performance or detection bias affect results?
4. **Measurement Quality**
- Are instruments validated and reliable?
- Are measures objective when possible, or subjective with acknowledged limitations?
- Is outcome assessment standardized?
- Are multiple measures used to triangulate findings?
**Reference:** See `references/scientific_method.md` for detailed principles and `references/experimental_design.md` for comprehensive design checklist.
### 2. Bias Detection
Identify and evaluate potential sources of bias that could distort findings.
**Apply when:**
- Reviewing published research
- Designing new studies
- Interpreting conflicting evidence
- Assessing research quality
**Systematic bias review:**
1. **Cognitive Biases (Researcher)**
- **Confirmation bias:** Are only supporting findings highlighted?
- **HARKing:** Were hypotheses stated a priori or formed after seeing results?
- **Publication bias:** Are negative results missing from literature?
- **Cherry-picking:** Is evidence selectively reported?
- Check for preregistration and analysis plan transparency
2. **Selection Biases**
- **Sampling bias:** Is sample representative of target population?
- **Volunteer bias:** Do participants self-select in systematic ways?
- **Attrition bias:** Is dropout differential between groups?
- **Survivorship bias:** Are only "survivors" visible in sample?
- Examine participant flow diagrams and compare baseline characteristics
3. **Measurement Biases**
- **Observer bias:** Could expectations influence observations?
- **Recall bias:** Are retrospective reports systematically inaccurate?
- **Social desirability:** Are responses biased toward acceptability?
- **Instrument bias:** Do measurement tools systematically err?
- Evaluate blinding, validation, and measurement objectivity
4. **Analysis Biases**
- **P-hacking:** Were multiple analyses conducted until significance emerged?
- **Outcome switching:** Were non-significant outcomes replaced with significant ones?
- **Selective reporting:** Are all planned analyses reported?
- **Subgroup fishing:** Were subgroup analyses conducted without correction?
- Check for study registration and compare to published outcomes
5. **Confounding**
- What variables could affect both exposure and outcome?
- Were confounders measured and controlled (statistically or by design)?
- Could unmeasured confounding explain findings?
- Are there plausible alternative explanations?
**Reference:** See `references/common_biases.md` for comprehensive bias taxonomy with detection and mitigation strategies.
### 3. Statistical Analysis Evaluation
Critically assess statistical methods, interpretation, and reporting.
**Apply when:**
- Reviewing quantitative research
- Evaluating data-driven claims
- Assessing clinical trial results
- Reviewing meta-analyses
**Statistical review checklist:**
1. **Sample Size and Power**
- Was a priori power analysis conducted?
- Is sample adequate for detecting meaningful effects?
- Is the study underpowered (common problem)?
- Do significant results from small samples raise flags for inflated effect sizes?
2. **Statistical Tests**
- Are tests appropriate for data type and distribution?
- Were test assumptions checked and met?
- Are parametric tests justified, or should non-parametric alternatives be used?
- Is the analysis matched to study design (e.g., paired vs. independent)?
3. **Multiple Comparisons**
- Were multiple hypotheses tested?
- Was correction applied (Bonferroni, FDR, other)?
- Are primary outcomes distinguished from secondary/exploratory?
- Could findings be false positives from multiple testing?
4. **P-Value Interpretation**
- Are p-values interpreted correctly (probability of data if null is true)?
- Is non-significance incorrectly interpreted as "no effect"?
- Is statistical significance conflated with practical importance?
- Are exact p-values reported, or only "p < .05"?
- Is there suspicious clustering just below .05?
5. **Effect Sizes and Confidence Intervals**
- Are effect sizes reported alongside significance?
- Are confidence intervals provided to show precision?
- Is the effect size meaningful in practical terms?
- Are standardized effect sizes interpreted with field-specific context?
6. **Missing Data**
- How much data is missing?
- Is missing data mechanism considered (MCAR, MAR, MNAR)?
- How is missing data handled (deletion, imputation, maximum likelihood)?
- Could missing data bias results?
7. **Regression and Modeling**
- Is the model overfitted (too many predictors, no cross-validation)?
- Are predictions made outside the data range (extrapolation)?
- Are multicollinearity issues addressed?
- Are model assumptions checked?
8. **Common Pitfalls**
- Correlation treated as causation
- Ignoring regression to the mean
- Base rate neglect
- Texas sharpshooter fallacy (pattern finding in noise)
- Simpson's paradox (confounding by subgroups)
**Reference:** See `references/statistical_pitfalls.md` for detailed pitfalls and correct practices.
### 4. Evidence Quality Assessment
Evaluate the strength and quality of evidence systematically.
**Apply when:**
- Weighing evidence for decisions
- Conducting literature reviews
- Comparing conflicting findings
- Determining confidence in conclusions
**Evidence evaluation framework:**
1. **Study Design Hierarchy**
- Systematic reviews/meta-analyses (highest for intervention effects)
- Randomized controlled trials
- Cohort studies
- Case-control studies
- Cross-sectional studies
- Case series/reports
- Expert opinion (lowest)
**Important:** Higher-level designs aren't always better quality. A well-designed observational study can be stronger than a poorly-conducted RCT.
2. **Quality Within Design Type**
- Risk of bias assessment (use appropriate tool: Cochrane RoB 2 for RCTs, ROBINS-I for non-randomized studies, Newcastle-Ottawa, etc.)
- Methodological rigor
- Transparency and reporting completeness
- Conflicts of interest
3. **GRADE Considerations (if applicable)**
- Start with design type (RCT = high, observational = low)
- **Provide conditional assessments:** "If X was done, then Y follows; if not, then Z is concern"
- **Note what additional information would resolve uncertainty**
## Reference Materials
This skill includes comprehensive reference materials that provide detailed frameworks for critical evaluation:
- **`references/scientific_method.md`** - Core principles of scientific methodology, the scientific process, critical evaluation criteria, red flags in scientific claims, causal inference standards, peer review, and open science principles
- **`references/common_biases.md`** - Comprehensive taxonomy of cognitive, experimental, methodological, statistical, and analysis biases with detection and mitigation strategies
- **`references/statistical_pitfalls.md`** - Common statistical errors and misinterpretations including p-value misunderstandings, multiple comparisons problems, sample size issues, effect size mistakes, correlation/causation confusion, regression pitfalls, and meta-analysis issues
- **`references/evidence_hierarchy.md`** - Traditional evidence hierarchy, GRADE system, study quality assessment criteria, domain-specific considerations, evidence synthesis principles, and practical decision frameworks
- **`references/logical_fallacies.md`** - Logical fallacies common in scientific discourse organized by type (causation, generalization, authority, relevance, structure, statistical) with examples and detection strategies
- **`references/experimental_design.md`** - Comprehensive experimental design checklist covering research questions, hypotheses, study design selection, variables, sampling, blinding, randomization, control groups, procedures, measurement, bias minimization, data management, statistical planning, ethical considerations, validity threats, and reporting standards
**When to consult references:**
- Load references into context when detailed frameworks are needed
- Use grep to search references for specific topics: `grep -r "pattern" references/`
- References provide depth; SKILL.md provides procedural guidance
- Consult references for comprehensive lists, detailed criteria, and specific examples
## Remember
**Scientific critical thinking is about:**
- Systematic evaluation using established principles
- Constructive critique that improves science
- Proportional confidence to evidence strength
- Transparency about uncertainty and limitations
- Consistent application of standards
- Recognition that all research has limitations
- Balance between skepticism and openness to evidence
**Always distinguish between:**
- Data (what was observed) and interpretation (what it means)
- Correlation and causation
- Statistical significance and practical importance
- Exploratory and confirmatory findings
- What is known and what is uncertain
- Evidence against a claim and evidence for the null
