|
|
@@ -0,0 +1,19 @@
|
|
|
+# Daily Log
|
|
|
+
|
|
|
+## Friday, June 14, 2024
|
|
|
+Relativly sedate day, mostly just rewriting the dataset system to ensure that loaded datasets use in evaluation are the same as the ones used in training. This ensures that there is no leakage of the train set into the validation and testing sets. Plan to meet with Ali and Brayden on Tuesday to go over results of the coverage analysis, since our results do not indicate the expected negative relationship between coverage and f1/accuracy/AUC.
|
|
|
+
|
|
|
+### Progress
|
|
|
+- Rewrote training to save datasets used - changed model directory structure to accomidate
|
|
|
+- Rewrote dataset loading in evaluation methods to ensure that the same datasets are used as in training
|
|
|
+- Trained new 10 epoch 30 model ensemble and evaluated
|
|
|
+ - Found better results than in previous ensembles, mostly negative relationship established, but still seeing sharp drop at very low coverage
|
|
|
+
|
|
|
+### Future
|
|
|
+- Investigate reasons for low accuracy at high certainty
|
|
|
+ - Do careful reading of papers on coverage analysis
|
|
|
+ - Compare to other models maybe??
|
|
|
+ - Meet with Ali and Brayden about results, brainstorm reasons
|
|
|
+- Continue reading more papers on uncertainty
|
|
|
+- Investigate BNNs further
|
|
|
+
|
