daily_log.md 2.7 KB
Relativly sedate day, mostly just rewriting the dataset system to ensure that loaded datasets use in evaluation are the same as the ones used in training. This ensures that there is no leakage of the train set into the validation and testing sets. Plan to meet with Ali and Brayden on Tuesday to go over results of the coverage analysis, since our results do not indicate the expected negative relationship between coverage and f1/accuracy/AUC.
- Rewrote training to save datasets used - changed model directory structure to accomidate
- Rewrote dataset loading in evaluation methods to ensure that the same datasets are used as in training
- Trained new 10 epoch 30 model ensemble and evaluated
- Found better results than in previous ensembles, mostly negative relationship established, but still seeing sharp drop at very low coverage
- Investigate reasons for low accuracy at high certainty
- Do careful reading of papers on coverage analysis
- Compare to other models maybe??
- Meet with Ali and Brayden about results, brainstorm reasons
- Continue reading more papers on uncertainty
- Investigate BNNs further
First day of Week 3! Pretty slow today as well, mainly working on ECE and figuring out why the coverage curves are so weird. Reading more about ensemble methods, going to try some calibration techniques. Overall, though, pretty good! Hope to be able to do some more after meeting with Ali and maybe Brayden, potentially look at some training methods (bagging, boosting etc.) and continue to investigate the weird coverage metrics.
- Implemented ECE metric
- Continued reading about uncertainty quantification
- Looked into bagging/boosting implementations
- Continue to investigate low accuracy at high certainty
- Continue reading
- Meet with Ali Wednesday
Slow day today, mostly continued with reading. Began training a new model with 100x30 runs/epochs, should be able to serve as base model for future work now that the data is deterministic
- Continued to read about calibration and calibration errors
- Investigated libraries for ensemble tranining
- Meet with Ali
- Continue reading
Work from home day, mostly planning and reading. Met with Ali and Brayden, discussed progress and began initial thinking towards future paths for project. Main focus on improving our uncertainty z
- Met with Ali and Brayden
- Began thinking about future progress
- Worked on ensemble a bit
- Fix uncertainty metric
- Evaluate performance of different metrics
- Write research proposal