On this page
Four different kinds of evidence
| Kind | What it can establish | What does not follow automatically |
|---|---|---|
| Software check | A program satisfies the tested rules and control cases. | That its model describes nature. |
| Exact finite calculation | A mathematical statement holds for the specified finite objects. | That it holds at every size or in a continuum limit. |
| Synthetic simulation | Behaviour occurs under the programmed model assumptions. | That real equipment exhibits the same effect. |
| Physical measurement | Data from a real setup support a limited measurement claim. | General validity without calibration, uncertainty analysis and repetition. |
An archived laboratory dataset comes from real measurements. Analysing it in this project is still not a new qualification of the project’s own equipment. An AI concept illustration is not a photograph of an assembled experiment.
Reading status labels
| Status | Meaning |
|---|---|
| PASS | The explicitly tested criteria were met. |
| FAIL | At least one specified criterion was violated. |
| INCONCLUSIVE | This test leaves the question undecided. |
| NOT_RUN / NOT_PERFORMED / NOT_EXECUTED | The relevant test was not executed. |
| INSUFFICIENT_COMMON_SUPPORT | Too few matching cases are available for the specified analysis. |
| MAIN / BRANCH_ONLY | Describes where source code is stored, not its scientific maturity. |
The precise meaning belongs to the relevant test contract. For example, a counterexample test may show PASS because an unsuitable filter was correctly rejected.
Reproduction is more than a checksum
A checksum such as SHA-256 helps detect changes to bytes. On its own, it proves neither the origin nor the correctness of a result. A replay runs the calculation again. A separately programmed checker reduces certain shared error risks; common authorship or libraries can still create dependencies.
External replication here means that other people repeat the result with their own scrutiny. Automated GitHub checks and internal counterchecks are useful, but do not replace that step.
- Read the precise claim and its limits.
- Record the source commit, dataset revision, random seed and software versions.
- Find the executed report, not just a planned test.
- Check that counterexamples and undecided cases were preserved.
- Compare the reported result with the criterion specified in advance.
What an uncertainty interval tells you
An interval describes uncertainty under a particular analysis model. If an interval for incremental benefit includes both negative and positive values, it does not establish an advantage. That is not proof of exact equality.
Paired comparisons analyse matching cases together. Some bootstrap intervals hold training and case selection fixed. They therefore do not additionally cover uncertainty from different training data, new generators or another selection process.