Merge branch 'system-rewrite' of https://git0.fmf.uni-lj.si/nschense/alzheimers_nn into system-rewrite

README.md

@@ -1,6 +1,10 @@
 # Alzheimers Diagnosis Neural Net Project Rewrite
 
-## TODO
-- [ ] Implement config saving for ensembles
-- [ ] Run more models
-- [ ] 
+This code is the current version for the Alzheimers CNN uncertanity estimation project. The project consists of a set of main scripts (in the root folder) and some utilities. In order to use the project:
+
+1. Edit "config.toml" with the details of the ensemble that you would like to train (size, name, epochs etc). Make sure that the model name and ensemble name are the same if you'd like to run the ensemble analysis later.
+2. Run "train_cnn.py". This will train and save a new ensemble of CNN models using the name and congfiguration options given in the config file.
+3. Run "ensemble_predict.py". This will generate the predictions of the models on the test and validation datasets and save them to the model ensemble folder. 
+4. Run "threshold_xarray.py". This run some analysis on the ensemble and generates a set of graphs and statistics. 
+
+"bayesian.py" is unfinished and does not currently work. The other two threshold files are old implementations. 'sensitivity_analysis.py' can be optionally used to generate some model number sensitivity data, but the implementation is buggy currently. Apologies for the messy code throughout! 

config.toml

@@ -6,18 +6,18 @@ xls_data = '/export/home/nschense/alzheimers/alzheimers_nn/LP_ADNIMERGE.csv'
 model_output = '/export/home/nschense/alzheimers/alzheimers_nn/saved_models/'
 
 [training]
-device = 'cuda:1'
-runs = 50
-max_epochs = 30
+device = 'cuda:1' 
+runs = 50 # Number of models
+max_epochs = 30 # Epochs per model
 
 [dataset]
-excluded_ids = [91, 108, 268, 269, 272, 279, 293, 296, 307]
+excluded_ids = [91, 108, 268, 269, 272, 279, 293, 296, 307] # Invalid images to exclude from the dataset
 validation_split = 0.4 #Splits the dataset into the train and validation/test set, 50% each for validation and test
 #|---TEST---|---VALIDATION---|---TRAIN---|
 #|splt*0.5  | split*0.5      | 1-split   |
 
 [model]
-name = 'cnn-50x30'
+name = 'cnn-50x30' # Ensemble name - will be folder for saved ensemble
 image_channels = 1
 clin_data_channels = 2
 
@@ -28,7 +28,7 @@ droprate = 0.5
 
 [operation]
 silent = false
-exclude_blank_ids = false
+exclude_blank_ids = false # Determines if invalid IDs are excluded or not
 
 [ensemble]
 name = 'cnn-50x30'

daily_log.md

@@ -1,57 +0,0 @@
-# Daily Log
-
-## Friday, June 14, 2024
-Relativly sedate day, mostly just rewriting the dataset system to ensure that loaded datasets use in evaluation are the same as the ones used in training. This ensures that there is no leakage of the train set into the validation and testing sets. Plan to meet with Ali and Brayden on Tuesday to go over results of the coverage analysis, since our results do not indicate the expected negative relationship between coverage and f1/accuracy/AUC. 
-
-### Progress
-- Rewrote training to save datasets used - changed model directory structure to accomidate
-- Rewrote dataset loading in evaluation methods to ensure that the same datasets are used as in training
-- Trained new 10 epoch 30 model ensemble and evaluated
-  - Found better results than in previous ensembles, mostly negative relationship established, but still seeing sharp drop at very low coverage
-
-### Future
-- Investigate reasons for low accuracy at high certainty
-  - Do careful reading of papers on coverage analysis
-  - Compare to other models maybe??
-  - Meet with Ali and Brayden about results, brainstorm reasons
-- Continue reading more papers on uncertainty
-- Investigate BNNs further
-
-## Monday, June 17, 2024
-First day of Week 3! Pretty slow today as well, mainly working on ECE and figuring out why the coverage curves are so weird. Reading more about ensemble methods, going to try some calibration techniques. Overall, though, pretty good! Hope to be able to do some more after meeting with Ali and maybe Brayden, potentially look at some training methods (bagging, boosting etc.) and continue to investigate the weird coverage metrics.
-
-### Progress
-- Implemented ECE metric
-- Continued reading about uncertainty quantification
-- Looked into bagging/boosting implementations
-
-### Future
-- Continue to investigate low accuracy at high certainty
-- Continue reading
-- Meet with Ali Wednesday
-
-
-### Tuesday, June 18, 2024
-Slow day today, mostly continued with reading. Began training a new model with 100x30 runs/epochs, should be able to serve as base model for future work now that the data is deterministic
-
-### Progress
-- Continued to read about calibration and calibration errors
-- Investigated libraries for ensemble tranining
-
-### Future
-- Meet with Ali
-- Continue reading
-
-## Wednesday, June 19, 2024
-Work from home day, mostly planning and reading. Met with Ali and Brayden, discussed progress and began initial thinking towards future paths for project. Main focus on improving our uncertainty
-z
-### Progress
-- Met with Ali and Brayden
-- Began thinking about future progress
-- Worked on ensemble a bit
-
-### Future
-- Fix uncertainty metric
-- Evaluate performance of different metrics
-- Write research proposal
-  

planning.md

@@ -1,29 +0,0 @@
-# Planning
-
-As of now, we have a program set up to be able to:
-
-- train an individual model with specific hyperparameters
-- train a ensemble of models with the identical hyperparameters
-- evaluate the accuracy of an ensemble of models
-- perform a coverage analysis on an ensemble of models
-
-The goal of this rewrite is to preserve those functions while making the program significantly cleaner and easier to use, and to make it easier to extend with new functionality in the future as well. The hope is for this project to take approximately ~1-2 days, and be completed by Monday (6/17). The additional features that I would like to implement are:
-
-- Recording sets of models and ensembles and reporting their metrics (must be able to distinguish between individual models and ensembles)
-- Better configurating settings, able to use waterfalling files (files in subdirectories override settings in main directory)
-- Descriptive configuration (describe the desired models, ensembles and analysis in a config file and have that be produced on program run)
-- Easier implementation of models and analysis (more use of classes and such)
-- Implementation of new metrics and ensembles
-- Deterministic dataloading (for a specified seed, the data used is set and does not change, even if the loading methods do)
-
-## Further Planning as of 7/8/24
-
-- With the implementation of uncertainty through standard deviation, confidence and entropy, next steps are
-  - Refactor current threshold implementation - very very messy and difficult to add new features
-  - Enable checking images for incorrect prediction, and predictions off of the main curve for stdev-conf curve thing
-  - Investigate physician confidence, and compare to uncertianty predictions
-  - Deep dive standard deviation
-  - Box plot?
-  - Investigate calibration - do we need it?
-  - Consider manuscript - should be thinking about writing
-