# Model Evaluation Code ## Description This folder should contain all the necessary code for the evaluation of the Bayeisan model and the Deep Ensemble. It should generate and save graphs and statistics for this. The included analyses should include - Performance information (i.e. basic information on accuracy, number correct, number incorrect, F1 score, etc.) - Some basic metrics for uncertainty (MCE, Brier) - Physican Confidence Analysis (graphing uncertainty vs physican confidence) - Longitudinal Analysis (graphing uncertainty on patients who remained stable CN, stable AD, or switched from CN to AD) - Noise introduction analysis (graphing uncertainty on normal image and images with increasing levels of Gaussian noise applied) "Uncertainty" should be taken to mean EITHER confidence or standard deviation for the ensemble and the bayesian network (i.e. the raw outputs distance from 0.5 or either the stdev from the Bayesian or the stdev of all the model outputs). Analyses should be evaluated with both. The code in the senior_research_thesis and the prebious alnn_rewrite/analysis folders should be consulted. The models should be loaded from the currently in-place config.toml ## Implementation Status The modular implementation now lives entirely in this folder. - All new source code is under `alnn_rewrite/analysis` - All generated artifacts are written under `alnn_rewrite/analysis_output` If a backend does not already have `model_evaluation_results.nc`, the pipeline now automatically evaluates that backend on the holdout datasets (validation + test) first, saves the generated netCDF into that backend's model output directory, and then runs the analyses. Uncertainty analyses are now run using both: - Confidence - Standard deviation (ensemble) - Predictive uncertainty (bayesian; computed via `bayesian_torch.utils.util.predictive_entropy`) ### Current Modules - `evaluate_models.py`: Orchestrator CLI for running selected analyses across ensemble and bayesian backends. - `defaults.py`: Centralized analysis defaults (threshold grid, noise factors, calibration bins, class index, decision threshold, bayesian MC passes). - `data_pipeline.py`: Shared analysis data loading and split/loader construction utilities. - `plotting.py`: Shared plotting functions for all analysis visualizations. - `uncertainty.py`: Shared confidence certainty/uncertainty transforms. - `model_utils.py`: Shared model behavior utilities (including bayesian sampling mode setup). - `runtime.py`: Runtime paths and JSON helpers. - `data_access.py`: netCDF loading, class probability extraction, and clinical table access. - `metrics.py`: Shared performance and calibration metrics. - `analysis_modules.py`: Performance, calibration, physician confidence, and longitudinal analyses. - `noise_analysis.py`: Evaluation-time Gaussian noise sensitivity analysis. ### How To Run From `alnn_rewrite`: ```bash python -m analysis.evaluate_models ``` Useful options: ```bash python -m analysis.evaluate_models \ --backend ensemble bayesian \ --run-name first_modular_run ``` If you want to skip noise analysis while validating the pipeline: ```bash python -m analysis.evaluate_models --skip-noise ``` If you want to quickly investigate longitudinal cohort breakdown only (without rerunning all analyses): ```bash python -m analysis.evaluate_models --longitudinal-breakdown-only ``` If you want to rerun only the noise uncertainty-vs-accuracy regression/correlation analysis from existing noise CSV outputs: ```bash python -m analysis.evaluate_models --noise-correlation-only --run-name run_YYYYMMDD_HHMMSS ``` If you want to generate only dataset composition documentation (total images, class counts, train/validation/test counts, and percentage breakdowns) without rerunning analyses: ```bash python -m analysis.evaluate_models --dataset-summary-only ``` This writes `dataset_summary.md` into the selected run directory under `analysis_output`. Most analysis tuning parameters are now intentionally centralized in `analysis/defaults.py` to reduce CLI verbosity and improve maintainability. ### Output Layout Each run creates a dedicated directory: ```text alnn_rewrite/analysis_output/ run_YYYYMMDD_HHMMSS/ run_manifest.json ensemble/ backend_summary.json plots_report.md performance_threshold_sweep.csv calibration_bins.csv performance_uncertainty_cutoff.csv performance_uncertainty_percentile_cutoff.csv physician_grouped_metrics.csv physician_confidence_grouped_metrics.csv physician_std_grouped_metrics.csv longitudinal_patient_summary.csv longitudinal_cohort_summary.csv longitudinal_uncertainty_by_cohort.csv longitudinal_confidence_patient_summary.csv longitudinal_std_patient_summary.csv longitudinal_confidence_cohort_summary.csv longitudinal_std_cohort_summary.csv noise_sensitivity.csv noise_accuracy_uncertainty_stats.csv noise_accuracy_uncertainty_summary.md plots/ performance_threshold_accuracy.png performance_threshold_f1.png calibration_reliability.png performance_uncertainty_cutoff_accuracy.png performance_uncertainty_cutoff_f1.png performance_uncertainty_percentile_cutoff_accuracy.png performance_uncertainty_percentile_cutoff_f1.png physician_confidence_boxplot.png physician_std_boxplot.png longitudinal_cohort_confidence.png longitudinal_cohort_std.png noise_sensitivity_accuracy.png noise_sensitivity_f1.png noise_confidence.png noise_standard_deviation.png (ensemble) / noise_predictive_uncertainty.png (bayesian) noise_accuracy_uncertainty_2d.png noise_examples/ ensemble_noise_examples.png bayesian_noise_examples.png ensemble_clean_scan_example.png bayesian_clean_scan_example.png bayesian/ ... (same structure) ``` The default noise schedule now includes noisier settings beyond the earlier small-sigma cases, so the saved example images show a clearer progression from lightly noised to heavily noised volumes. Performance analysis now includes two uncertainty cutoff views: - A raw-value cutoff sweep that keeps samples with uncertainty below the percentile-derived cutoff value. - A percentile-ranked cutoff sweep that is plotted from least restricted (left, all samples) to most restricted (right, lowest-uncertainty subset). Noise analysis now saves individual series plots (accuracy, F1, confidence, and standard deviation or predictive uncertainty) so each metric is shown on its own graph. Noise plots now label the x-axis as Gaussian Noise Factor to reflect that values are multipliers on the actual noise sigma.