alnn_rewrite/final_analysis_thoughs.md

Final Analysis Thoughts

Why focus on Ensemble?

Currently, the project consists of both an ensemble component and a bayesian model component. Mostly the same expirements are run on both, but they produce very different results. The ensemble shows very good accuracy (a marked improvement over a single model) as well as promise with regards to uncertainty quantitifcation (although performance compared to a single model still needs to be evaluated). On the other hand, while the bayesian model does seem to do well with uncertainty quantification (from a monotonicity perspective), it struggles majorly with the actual predictions, and has a pitiful overall accuracy. This, in addition to the additional work required to get the ensemble in a useable state, means that the overall value of the bayesian analysis is, in my opinion, rather limited.

In contrast, ensemble methods have a couple of avenues of research still left to investigate, and show overall more promise as a pos-hoc method and as something that is very easy to implement into a pipeline. For these reasons, I think in a project focusing on clinical use, it makes sense to focus on the ensemble, and evaluate how well that method improves accuracy and uncertainty quantification compared to a single model.

Next Steps

• Rewrite the analysis pipeline - strip out the fluff AI code, make the analyses more concise (keeping bayesian for now)
  • Make sure to add in the modified dnn_to_bnn() function into our code for reliability
• Convert to ensemble only and remove bayesian analysis
• Add in more ensemble specific analyeses (sensitivity to model numbers, pruning performance increases, etc.)

Final Analysis Lineup

If removing the bayesian ends up being the route that we actually want to do, the following lineup for the final paper makes the most sense to me

• Ensemble performance vs. single model
• Sensitivity of ensemble size
• Pruining performance increase with ensemble size
• Ensemble UQ performance (i.e. coverage plot
• Ensemble UQ vs single model UQ
• Noise resiliance vs single model
• Noise UQ (fitting to the noise-uncertainty relationship)
• Scatter of confidence vs stdev (investigating weird stdev performance)
• In-depth investigation of stdev

I think this is a pretty solid lineup, but I'll continue to work on it.