Adaptive Sampling

Adaptive Sampling lets Sailfish stop collecting samples automatically once your results are statistically stable. This reduces test time while maintaining rigor, and removes the need to guess a fixed sample size up front.

Why it exists

Fixed sample sizes are either too small (noisy results) or too large (wasted time). Adaptive sampling balances speed and precision by collecting “just enough” data for reliable comparisons.

How it works

After warmups, Sailfish collects iterations and evaluates convergence when there are at least MinimumSampleSize samples. On each check:

• Compute CV = standard deviation / mean
• Compute the relative CI width at the configured ConfidenceLevel
• If both are within thresholds, sampling stops; otherwise continue until MaximumSampleSize

See also: Confidence Intervals

See also: Precision/Time Budget Controller

Defaults and tunables

• MinimumSampleSize: 10
• MaximumSampleSize: 1000 (safety cap)
• TargetCoefficientOfVariation: 0.05 (5%)
• ConfidenceLevel: 0.95 (95%)
• MaxConfidenceIntervalWidth: 0.20 (20% relative CI width)
• UseRelativeConfidenceInterval: true

Parameter selection (pilot phase)

Sailfish runs a short pilot (the minimum phase) and classifies your method by speed using the median iteration time. It then recommends local thresholds for the rest of the run:

• UltraFast (< 50 µs): MinN ≥ 50; TargetCV ≥ max(user, 3%); MaxCI ≤ min(user, 12%)
• Fast (< 0.5 ms): MinN ≥ 30; TargetCV ≥ max(user, 4%); MaxCI ≤ min(user, 15%)
• Medium (< 5 ms): MinN ≥ 20; TargetCV ≥ max(user, 5%); MaxCI ≤ min(user, 20%)
• Slow (< 50 ms): MinN ≥ 15; TargetCV ≥ max(user, 7%); MaxCI ≤ min(user, 25%)
• VerySlow (≥ 50 ms): MinN ≥ 10; TargetCV ≥ max(user, 10%); MaxCI ≤ min(user, 30%)

Notes:

• Safety rules: never tighten CV below what you requested; never exceed your MaxCI budget.
• The recommended MinN is applied as a local floor only for convergence checks; your configured MinimumSampleSize still controls the initial pilot (minimum phase).
• If classification fails, Sailfish falls back to your configured thresholds.

How to enable

Per-class (attribute-based)

[Sailfish(UseAdaptiveSampling = true, TargetCoefficientOfVariation = 0.05, MaximumSampleSize = 1000)]
public class StableTiming
{
    [SailfishMethod]
    public async Task Work() => await Task.Delay(10);
}

Globally (all tests in a run)

var runSettings = RunSettingsBuilder.CreateBuilder()
    .WithGlobalAdaptiveSampling(targetCoefficientOfVariation: 0.05, maximumSampleSize: 500)
    .Build();

Interactions with fixed settings

• SampleSize on the attribute still works for fixed sampling
• In an adaptive run, SampleSizeOverride (from run settings) is treated as the MaximumSampleSize
• Warmups (NumWarmupIterations) still apply

When to use

Use adaptive sampling when:

• You want consistent precision across tests without hand-tuning N
• CI runtime matters and you want tests to stop early when stable
• Your benchmarks vary in noise; stable ones converge fast, noisy ones gather more data

Prefer fixed sampling when:

• You must replicate historical runs with exact N (compliance/audit)
• You’re comparing runs across time where identical N is a hard requirement

Migration (existing projects)

Adaptive sampling is backward-compatible and opt-in.

• Do nothing: fixed sampling continues as before
• Enable per class: UseAdaptiveSampling = true
• Or enable globally with the builder (recommended for consistent policy)
• If using SampleSizeOverride, note it caps the number of adaptive samples

Best practices

• Tighten TargetCoefficientOfVariation (e.g., 0.02) for highly stable microbenchmarks
• Raise MaximumSampleSize for noisy workloads to allow convergence
• Keep MinimumSampleSize ≥ 10 so CI estimates are meaningful
• In CI, prefer class attributes for clarity and apply global defaults to unify policy

Troubleshooting

• “Never converges”: lower thresholds or raise MaximumSampleSize; investigate outliers
• “Too precise to be true”: check for caching/mocks; extremely low variance can be misleading
• “Runs are slow”: relax thresholds or reduce MaximumSampleSize

Internals (for contributors)

• Strategy selection happens in TestCaseIterator (adaptive vs fixed)
• Convergence detection is implemented in StatisticalConvergenceDetector
• Execution settings are composed from attributes plus global overrides via ExecutionExtensionMethods.RetrieveExecutionTestSettings(...)
• Global overrides flow from RunSettingsBuilder.WithGlobalAdaptiveSampling(...) → IRunSettings → RetrieveExecutionTestSettings(...)