alnn_rewrite/alnn_rewrite.log

2026-06-30 18:47:08,175 - ERROR - Pipeline failed: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 227, in run_background_pipeline
    task_func(self.update_progress, config)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
2026-06-30 18:47:12,955 - ERROR - Pipeline failed: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 227, in run_background_pipeline
    task_func(self.update_progress, config)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
2026-06-30 18:54:45,258 - ERROR - Pipeline failed: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 227, in run_background_pipeline
    task_func(self.update_progress, config)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
2026-06-30 19:08:20,961 - ERROR - Pipeline failed: App is not running
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 332, in run_background_pipeline
    task_func(self.update_progress, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 66, in dummy_task
    progress_callback(advance=1)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 381, in update_progress
    self.call_from_thread(_update)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/textual/app.py", line 1819, in call_from_thread
    raise RuntimeError("App is not running")
RuntimeError: App is not running
2026-06-30 19:18:14,215 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 19:22:46,181 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 21:49:24,173 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 21:58:23,232 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:02:17,566 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:02:24,454 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:05:28,118 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:06:52,271 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:07:09,800 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:07:16,120 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:07:25,317 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:08:39,971 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:08:44,617 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:09:01,476 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:09:52,766 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:14:38,011 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:15:31,941 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:19:35,748 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:25:58,294 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:26:28,903 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user while paused.
2026-06-30 22:37:54,542 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user while paused.
2026-06-30 22:38:05,493 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user while paused.
2026-06-30 23:00:47,768 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 23:09:29,204 - INFO - [SYSTEM] Application initialized and ready.
2026-06-30 23:13:16,084 - INFO - [SYSTEM] Application initialized and ready.
2026-06-30 23:13:17,733 - ERROR - [SYSTEM] No scenario selected! Please select a scenario from the dropdown.
2026-06-30 23:13:20,804 - INFO - [SYSTEM] Initiating Scenario: 2. Load, Evaluate, & Noise Analysis
2026-06-30 23:13:20,805 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:13:20,807 - INFO - [SYSTEM] Starting Task 1/3: Load Image and ADNIMERGE
2026-06-30 23:13:20,807 - ERROR - Pipeline failed: 'log'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 293, in run_background_pipeline
    task_func(task_tracker, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/tasks.py", line 10, in dummy_task
    log = state["log"]
          ~~~~~^^^^^^^
KeyError: 'log'
2026-06-30 23:13:20,808 - ERROR - [SYSTEM] ERROR: 'log'
2026-06-30 23:13:33,900 - INFO - [SYSTEM] Initiating Scenario: 2. Load, Evaluate, & Noise Analysis
2026-06-30 23:13:33,901 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:13:33,903 - INFO - [SYSTEM] Starting Task 1/3: Load Image and ADNIMERGE
2026-06-30 23:13:33,904 - ERROR - Pipeline failed: 'log'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 293, in run_background_pipeline
    task_func(task_tracker, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/tasks.py", line 10, in dummy_task
    log = state["log"]
          ~~~~~^^^^^^^
KeyError: 'log'
2026-06-30 23:13:33,904 - ERROR - [SYSTEM] ERROR: 'log'
2026-06-30 23:13:38,868 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-06-30 23:13:38,869 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:13:38,871 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-06-30 23:13:38,872 - ERROR - Pipeline failed: 'log'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 293, in run_background_pipeline
    task_func(task_tracker, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/tasks.py", line 10, in dummy_task
    log = state["log"]
          ~~~~~^^^^^^^
KeyError: 'log'
2026-06-30 23:13:38,872 - ERROR - [SYSTEM] ERROR: 'log'
2026-06-30 23:15:39,501 - INFO - [SYSTEM] Application initialized and ready.
2026-06-30 23:15:42,574 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-06-30 23:15:42,574 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:15:42,576 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-06-30 23:15:42,577 - INFO - [Load Image and ADNIMERGE] Initializing process...
2026-06-30 23:15:44,200 - INFO - [Load Image and ADNIMERGE] Halfway through current task execution...
2026-06-30 23:15:45,290 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-06-30 23:15:45,300 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-06-30 23:15:45,302 - INFO - [Train Regular Models] Initializing process...
2026-06-30 23:15:46,918 - INFO - [Train Regular Models] Halfway through current task execution...
2026-06-30 23:15:48,003 - INFO - [Train Regular Models] Task completed.
2026-06-30 23:15:48,004 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-06-30 23:15:48,005 - INFO - [Train Bayesian Models] Initializing process...
2026-06-30 23:15:49,636 - INFO - [Train Bayesian Models] Halfway through current task execution...
2026-06-30 23:15:50,716 - INFO - [Train Bayesian Models] Task completed.
2026-06-30 23:15:50,718 - INFO - [SYSTEM] Starting Task 4/5: Evaluate Regular Models
2026-06-30 23:15:50,719 - INFO - [Evaluate Regular Models] Initializing process...
2026-06-30 23:15:52,346 - INFO - [Evaluate Regular Models] Halfway through current task execution...
2026-06-30 23:15:53,414 - INFO - [Evaluate Regular Models] Task completed.
2026-06-30 23:15:53,416 - INFO - [SYSTEM] Starting Task 5/5: Evaluate Bayesian Models
2026-06-30 23:15:53,417 - INFO - [Evaluate Bayesian Models] Initializing process...
2026-06-30 23:15:53,671 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-06-30 23:15:53,685 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 20:54:06,351 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 20:54:46,133 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:01:54,340 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:03:07,022 - INFO - [SYSTEM] Copied default config to ./outputs/test1/config.toml. Ready for editing!
2026-07-07 21:04:00,870 - INFO - [SYSTEM] Successfully loaded config from ./outputs/test1/config.toml
2026-07-07 21:23:28,025 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:23:35,938 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 21:23:42,961 - ERROR - [SYSTEM] Invalid or missing scenario 'None' in config.toml!
2026-07-07 21:56:46,299 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:57:01,122 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 21:57:09,504 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 21:57:09,515 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 21:57:09,516 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 21:57:09,517 - ERROR - Pipeline failed: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 254, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 118, in load_adni_data_from_file
    xls_values = xls_preprocessor(pd.read_csv(xls_file))  # type: ignore
                                  ~~~~~~~~~~~^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 873, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 300, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1645, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1904, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/common.py", line 930, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 21:57:09,518 - ERROR - [SYSTEM] ERROR: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 21:57:20,610 - ERROR - [SYSTEM] Pipeline start cancelled by user (folder non-empty).
2026-07-07 21:58:55,263 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:59:21,987 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:59:23,885 - ERROR - [SYSTEM] Please specify a working directory.
2026-07-07 21:59:29,273 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 21:59:30,989 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 21:59:30,992 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 21:59:30,993 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 21:59:30,993 - ERROR - Pipeline failed: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 257, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 118, in load_adni_data_from_file
    xls_values = xls_preprocessor(pd.read_csv(xls_file))  # type: ignore
                                  ~~~~~~~~~~~^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 873, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 300, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1645, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1904, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/common.py", line 930, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 21:59:30,995 - ERROR - [SYSTEM] ERROR: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 22:00:48,869 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:00:54,847 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:00:56,988 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:00:56,991 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:00:56,992 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:01:00,884 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:01:00,888 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:01:00,889 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:01:00,890 - INFO - [Train Regular Models] Initializing process...
2026-07-07 22:01:02,509 - INFO - [Train Regular Models] Halfway through current task execution...
2026-07-07 22:01:03,576 - INFO - [Train Regular Models] Task completed.
2026-07-07 22:01:03,577 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-07-07 22:01:03,578 - INFO - [Train Bayesian Models] Initializing process...
2026-07-07 22:01:04,289 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:01:04,321 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:40:31,759 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:40:40,591 - ERROR - [SYSTEM] Failed to initialize directory/config: [Errno 13] Permission denied: '/outputs'
2026-07-07 22:40:49,252 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-07 22:40:55,338 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:40:55,348 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:40:55,348 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:40:58,312 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:40:58,312 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:40:58,314 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:40:58,314 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:40:58,315 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:41:14,253 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 38.0169, Val Loss: 46.0747, Train Acc: 0.5034, Val Acc: 0.5034
2026-07-07 22:41:27,881 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 29.5272, Val Loss: 41.2137, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 22:41:30,945 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:41:41,559 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 25.7706, Val Loss: 35.2926, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 22:41:41,792 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 22:41:45,691 - INFO - [SYSTEM] Pipeline Resumed.
2026-07-07 22:41:50,686 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 22:41:55,302 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 24.3820, Val Loss: 38.6998, Train Acc: 0.5906, Val Acc: 0.5906
2026-07-07 22:42:09,063 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 21.6774, Val Loss: 39.5588, Train Acc: 0.6040, Val Acc: 0.6040
2026-07-07 22:42:22,836 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 20.0827, Val Loss: 41.8771, Train Acc: 0.5772, Val Acc: 0.5772
2026-07-07 22:42:36,641 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 18.2864, Val Loss: 37.1265, Train Acc: 0.5772, Val Acc: 0.5772
2026-07-07 22:42:36,744 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 22:42:50,611 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 18.5486, Val Loss: 39.2183, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 22:43:04,475 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 16.0196, Val Loss: 32.6778, Train Acc: 0.6577, Val Acc: 0.6577
2026-07-07 22:43:18,348 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 15.5705, Val Loss: 31.4642, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 22:43:32,211 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 14.0887, Val Loss: 45.1521, Train Acc: 0.5638, Val Acc: 0.5638
2026-07-07 22:45:59,606 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:46:09,907 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:46:11,087 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:46:11,099 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:46:11,100 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:46:13,818 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:46:13,823 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:46:13,824 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:46:13,824 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:46:13,825 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:46:13,826 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:46:28,132 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 35.8615, Val Loss: 39.1921, Train Acc: 0.5235, Val Acc: 0.5235
2026-07-07 22:46:36,897 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:46:37,458 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:47:11,214 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:47:11,217 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:47:11,217 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:47:13,919 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:47:13,920 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:47:13,925 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:47:13,925 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:47:13,926 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:47:13,927 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:47:27,633 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 31.7686, Val Loss: 35.9996, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 22:47:30,481 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:47:30,735 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:47:32,317 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:47:32,321 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:47:32,322 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:47:35,102 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:47:35,107 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:47:35,108 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:47:35,109 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:47:35,109 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:47:35,110 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:47:48,903 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 35.5566, Val Loss: 51.9955, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:48:02,651 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 27.3013, Val Loss: 55.1044, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:48:16,422 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 21.5175, Val Loss: 51.3943, Train Acc: 0.4832, Val Acc: 0.4832
2026-07-07 22:48:30,259 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 20.9605, Val Loss: 49.0898, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 22:48:44,110 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 18.0304, Val Loss: 46.6057, Train Acc: 0.4765, Val Acc: 0.4765
2026-07-07 22:48:57,963 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 16.3244, Val Loss: 49.9473, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:49:11,819 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 15.5546, Val Loss: 53.5298, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:49:11,915 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 22:49:25,775 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 14.0157, Val Loss: 35.8735, Train Acc: 0.6107, Val Acc: 0.6107
2026-07-07 22:49:39,632 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 12.6665, Val Loss: 32.9891, Train Acc: 0.7181, Val Acc: 0.7181
2026-07-07 22:49:53,492 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 12.5009, Val Loss: 35.4068, Train Acc: 0.6107, Val Acc: 0.6107
2026-07-07 22:50:07,350 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 10.9609, Val Loss: 29.3653, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 22:50:21,206 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 11.1512, Val Loss: 38.1878, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 22:50:35,064 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 9.2618, Val Loss: 30.5188, Train Acc: 0.7315, Val Acc: 0.7315
2026-07-07 22:50:48,927 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 10.1587, Val Loss: 47.8058, Train Acc: 0.5235, Val Acc: 0.5235
2026-07-07 22:50:49,018 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 22:51:02,881 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 8.3822, Val Loss: 35.1154, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 22:51:16,742 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 9.1352, Val Loss: 36.4209, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 22:51:30,606 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 8.3755, Val Loss: 45.6383, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 22:51:44,471 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 7.2167, Val Loss: 30.6616, Train Acc: 0.7047, Val Acc: 0.7047
2026-07-07 22:51:58,333 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 7.5580, Val Loss: 34.0123, Train Acc: 0.6174, Val Acc: 0.6174
2026-07-07 22:52:12,203 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 6.0840, Val Loss: 64.6364, Train Acc: 0.4966, Val Acc: 0.4966
2026-07-07 22:52:12,253 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:52:26,061 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 8.6929, Val Loss: 40.2682, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 22:52:26,150 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 22:52:37,622 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:52:39,562 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:52:39,564 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:52:39,565 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:52:40,028 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 6.6049, Val Loss: 37.4700, Train Acc: 0.5906, Val Acc: 0.5906
2026-07-07 22:52:42,067 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 252, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:52:42,068 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:52:53,931 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 6.0106, Val Loss: 30.1979, Train Acc: 0.7315, Val Acc: 0.7315
2026-07-07 22:52:57,540 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:52:57,542 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:52:57,543 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:53:00,126 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 252, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:53:00,126 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:53:59,356 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:53:59,358 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:53:59,359 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:54:01,108 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 252, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:54:01,108 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:56:14,146 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:56:27,820 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:56:31,532 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:56:31,535 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:56:31,535 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:56:34,443 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:56:34,447 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:56:34,448 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:56:34,449 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:56:34,449 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:56:34,450 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:56:48,915 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 31.9007, Val Loss: 40.0863, Train Acc: 0.5302, Val Acc: 0.5302
2026-07-07 22:56:55,874 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:56:55,915 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:59:41,802 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:59:51,565 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:59:53,442 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:59:53,445 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:59:53,445 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:59:56,393 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:59:56,397 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:59:56,398 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:59:56,399 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:59:56,399 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:59:56,400 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:00:10,718 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 30.2033, Val Loss: 40.4815, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 23:00:16,121 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:00:16,157 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:00:19,835 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:00:19,836 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:00:19,837 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:00:22,440 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:00:22,444 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:00:22,445 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:00:22,446 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:00:22,446 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:00:22,447 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:00:27,800 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:00:27,946 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:00:56,651 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:00:56,653 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:00:56,654 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:00:58,796 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:00:58,800 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:00:58,801 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:00:58,801 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:00:58,802 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:00:58,802 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:01:12,503 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 33.8031, Val Loss: 38.0120, Train Acc: 0.7919, Val Acc: 0.7919
2026-07-07 23:01:26,219 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 27.8724, Val Loss: 40.8105, Train Acc: 0.4966, Val Acc: 0.4966
2026-07-07 23:01:39,984 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 25.4381, Val Loss: 35.0456, Train Acc: 0.6644, Val Acc: 0.6644
2026-07-07 23:01:53,781 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 23.3444, Val Loss: 35.3899, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 23:02:07,612 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 21.0827, Val Loss: 33.5305, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 23:02:21,436 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 20.1785, Val Loss: 37.8223, Train Acc: 0.7517, Val Acc: 0.7517
2026-07-07 23:02:35,268 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 18.5837, Val Loss: 43.5106, Train Acc: 0.5168, Val Acc: 0.5168
2026-07-07 23:02:35,344 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:02:38,077 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:02:38,490 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:05:49,105 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:06:05,829 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-07 23:06:08,440 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:06:08,451 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:06:08,451 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:06:11,353 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:06:11,357 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:06:11,358 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:06:11,359 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:06:11,359 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:06:11,360 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:06:19,433 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:06:19,831 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:07:23,454 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:07:23,457 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:07:23,457 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:07:26,158 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:07:26,162 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:07:26,163 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:07:26,164 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:07:26,164 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:07:26,165 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:07:39,894 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 36.3690, Val Loss: 39.1494, Train Acc: 0.6242, Val Acc: 0.6242
2026-07-07 23:07:47,496 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 23:07:49,550 - INFO - [SYSTEM] Pipeline Resumed.
2026-07-07 23:07:52,492 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 23:07:54,747 - INFO - [SYSTEM] Pipeline Resumed.
2026-07-07 23:07:57,610 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 28.3761, Val Loss: 42.0669, Train Acc: 0.5034, Val Acc: 0.5034
2026-07-07 23:07:59,573 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:07:59,949 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:08:44,258 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:08:55,979 - INFO - [SYSTEM] Copied default config to ./outputs/exp2/config.toml. Ready for editing!
2026-07-07 23:09:00,092 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:09:00,094 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:09:00,095 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:09:02,563 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:02,564 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:11,258 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:09:11,272 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:09:11,273 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:09:13,452 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:13,453 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:50,869 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:09:58,158 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp2/config.toml
2026-07-07 23:09:59,995 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:09:59,998 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:09:59,998 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:10:02,945 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:10:02,950 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:10:02,951 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:10:02,952 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:10:02,952 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp2/normal_models/intermediate_models
2026-07-07 23:10:02,953 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:10:17,381 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 34.3198, Val Loss: 37.7880, Train Acc: 0.7450, Val Acc: 0.7450
2026-07-07 23:10:31,134 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 28.2604, Val Loss: 35.1542, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 23:10:44,929 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 25.2059, Val Loss: 37.1065, Train Acc: 0.6443, Val Acc: 0.6443
2026-07-07 23:10:58,741 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 23.2649, Val Loss: 37.7584, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 23:11:12,573 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 20.9050, Val Loss: 36.4091, Train Acc: 0.6040, Val Acc: 0.6040
2026-07-07 23:11:26,413 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 19.2700, Val Loss: 36.3774, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 23:11:40,255 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 18.0514, Val Loss: 35.6015, Train Acc: 0.6376, Val Acc: 0.6376
2026-07-07 23:11:40,355 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:11:54,199 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 17.0273, Val Loss: 34.6692, Train Acc: 0.6309, Val Acc: 0.6309
2026-07-07 23:12:08,042 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 16.3381, Val Loss: 56.3836, Train Acc: 0.5369, Val Acc: 0.5369
2026-07-07 23:12:21,887 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 17.3398, Val Loss: 38.8979, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 23:12:35,737 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 14.4686, Val Loss: 30.7893, Train Acc: 0.8054, Val Acc: 0.8054
2026-07-07 23:12:49,582 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 13.8248, Val Loss: 35.9297, Train Acc: 0.6242, Val Acc: 0.6242
2026-07-07 23:13:03,427 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 13.3095, Val Loss: 35.1133, Train Acc: 0.7181, Val Acc: 0.7181
2026-07-07 23:13:17,272 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 12.7092, Val Loss: 44.7456, Train Acc: 0.5638, Val Acc: 0.5638
2026-07-07 23:13:17,359 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 23:13:31,211 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 13.0241, Val Loss: 49.0018, Train Acc: 0.5503, Val Acc: 0.5503
2026-07-07 23:13:45,062 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 11.3344, Val Loss: 40.5898, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 23:13:58,917 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 10.7968, Val Loss: 33.3214, Train Acc: 0.6577, Val Acc: 0.6577
2026-07-07 23:14:12,766 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 9.2847, Val Loss: 32.2253, Train Acc: 0.7919, Val Acc: 0.7919
2026-07-07 23:14:26,619 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 9.5039, Val Loss: 28.7371, Train Acc: 0.8523, Val Acc: 0.8523
2026-07-07 23:14:40,468 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 8.9888, Val Loss: 35.7847, Train Acc: 0.6107, Val Acc: 0.6107
2026-07-07 23:14:54,317 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 9.0884, Val Loss: 32.0707, Train Acc: 0.6644, Val Acc: 0.6644
2026-07-07 23:14:54,411 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 23:15:08,264 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 8.4898, Val Loss: 38.5546, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 23:15:22,114 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 7.8137, Val Loss: 33.0331, Train Acc: 0.7114, Val Acc: 0.7114
2026-07-07 23:15:35,966 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 7.0856, Val Loss: 27.0260, Train Acc: 0.8054, Val Acc: 0.8054
2026-07-07 23:15:49,811 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 7.6837, Val Loss: 45.5512, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 23:16:03,659 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 6.6017, Val Loss: 32.3892, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 23:16:17,508 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 7.7363, Val Loss: 34.3906, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 23:16:31,357 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 7.1077, Val Loss: 41.1836, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 23:16:31,445 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-07 23:16:45,297 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 6.1704, Val Loss: 38.0425, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 23:16:59,145 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 6.1988, Val Loss: 29.8566, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 23:16:59,362 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 96, in train_normal_task
    test_loss, test_acc = tn.test_model(
                          ~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<5 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 65, in test_model
    outputs = model((mri, xls))
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 66, in forward
    image_out = self.image_section(image_data)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 36, in forward
    x = self.midflow(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 91, in forward
    a = nn.ELU()(self.block(x) + x)
                 ~~~~~~~~~~~~~~^~~
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:16:59,370 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:18:31,229 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:18:31,236 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:18:31,237 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:18:34,007 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:18:34,009 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:18:34,010 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:18:34,011 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:18:34,011 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp2/normal_models/intermediate_models
2026-07-07 23:18:34,012 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:18:39,480 - ERROR - Pipeline failed: App is not running
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 81, in train_normal_task
    model, history = tn.train_model(
                     ~~~~~~~~~~~~~~^
        log=log,
        ^^^^^^^^
    ...<9 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 190, in train_model
    train_loss, val_loss, train_acc, val_acc = train_epoch(
                                               ~~~~~~~~~~~^
        model,
        ^^^^^^
    ...<7 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 120, in train_epoch
    train_progress.update(total=len(train_loader), advance=1)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/progress.py", line 64, in update
    self.app.call_from_thread(_do_update)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/textual/app.py", line 1819, in call_from_thread
    raise RuntimeError("App is not running")
RuntimeError: App is not running
2026-07-07 23:18:39,481 - ERROR - [SYSTEM] ERROR: App is not running
2026-07-07 23:22:18,712 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:22:30,814 - INFO - [SYSTEM] Copied default config to ./outputs/experiment_1/config.toml. Ready for editing!
2026-07-07 23:22:33,000 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:22:33,003 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:22:33,004 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:22:35,714 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:22:35,718 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:22:35,720 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:22:35,720 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:22:35,721 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/experiment_1/normal_models/intermediate_models
2026-07-07 23:22:35,721 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:22:50,086 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 37.1905, Val Loss: 42.7157, Train Acc: 0.5034, Val Acc: 0.5034
2026-07-07 23:23:03,790 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 25.8338, Val Loss: 35.7036, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 23:23:17,550 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 21.3707, Val Loss: 45.1811, Train Acc: 0.4765, Val Acc: 0.4765
2026-07-07 23:23:31,351 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 19.2528, Val Loss: 36.2448, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 23:23:45,172 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 16.2027, Val Loss: 32.4055, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 23:23:58,997 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 14.6334, Val Loss: 30.1882, Train Acc: 0.8255, Val Acc: 0.8255
2026-07-07 23:24:06,489 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:24:06,847 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:27:52,571 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:27:59,544 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 23:28:01,690 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:28:01,701 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:28:01,701 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:28:04,661 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:28:04,663 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:28:04,668 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:28:04,668 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:28:04,669 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:28:04,669 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:28:18,992 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5685, Val Loss: 0.7230, Train Acc: 0.7034, Val Acc: 0.4631
2026-07-07 23:28:32,736 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4527, Val Loss: 0.7075, Train Acc: 0.8365, Val Acc: 0.4966
2026-07-07 23:28:46,493 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.3957, Val Loss: 0.5646, Train Acc: 0.9125, Val Acc: 0.8188
2026-07-07 23:29:00,299 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3627, Val Loss: 0.7107, Train Acc: 0.9506, Val Acc: 0.5101
2026-07-07 23:29:14,125 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3393, Val Loss: 0.6061, Train Acc: 0.9715, Val Acc: 0.7181
2026-07-07 23:29:27,967 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.3099, Val Loss: 0.6734, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-07 23:29:41,808 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2903, Val Loss: 0.6156, Train Acc: 0.9810, Val Acc: 0.5973
2026-07-07 23:29:41,901 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:29:55,752 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2748, Val Loss: 0.6656, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-07 23:30:09,598 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2596, Val Loss: 0.5948, Train Acc: 0.9848, Val Acc: 0.6242
2026-07-07 23:30:23,445 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2503, Val Loss: 0.5690, Train Acc: 0.9791, Val Acc: 0.7987
2026-07-07 23:30:37,293 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.2408, Val Loss: 0.8184, Train Acc: 0.9829, Val Acc: 0.5235
2026-07-07 23:30:51,140 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.2546, Val Loss: 0.6428, Train Acc: 0.9696, Val Acc: 0.5906
2026-07-07 23:31:04,990 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.2260, Val Loss: 0.6634, Train Acc: 0.9848, Val Acc: 0.5839
2026-07-07 23:31:18,836 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.2089, Val Loss: 0.6569, Train Acc: 0.9829, Val Acc: 0.5772
2026-07-07 23:31:18,924 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 23:31:32,776 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1820, Val Loss: 0.5285, Train Acc: 0.9924, Val Acc: 0.7651
2026-07-07 23:31:46,627 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1814, Val Loss: 0.5080, Train Acc: 0.9848, Val Acc: 0.8054
2026-07-07 23:32:00,481 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1848, Val Loss: 0.5403, Train Acc: 0.9810, Val Acc: 0.8255
2026-07-07 23:32:14,331 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1725, Val Loss: 0.5621, Train Acc: 0.9867, Val Acc: 0.6577
2026-07-07 23:32:28,183 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1524, Val Loss: 0.8701, Train Acc: 0.9848, Val Acc: 0.5034
2026-07-07 23:32:42,035 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1537, Val Loss: 0.6627, Train Acc: 0.9848, Val Acc: 0.5906
2026-07-07 23:32:55,889 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1386, Val Loss: 0.5572, Train Acc: 0.9829, Val Acc: 0.6510
2026-07-07 23:32:55,982 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 23:33:09,838 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1314, Val Loss: 0.5282, Train Acc: 0.9867, Val Acc: 0.7919
2026-07-07 23:33:23,693 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.1305, Val Loss: 0.4771, Train Acc: 0.9848, Val Acc: 0.8523
2026-07-07 23:33:37,549 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1217, Val Loss: 0.7991, Train Acc: 0.9848, Val Acc: 0.5436
2026-07-07 23:33:51,407 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.1192, Val Loss: 0.6912, Train Acc: 0.9867, Val Acc: 0.5503
2026-07-07 23:34:05,268 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.1029, Val Loss: 0.6175, Train Acc: 0.9886, Val Acc: 0.6443
2026-07-07 23:34:19,129 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.1157, Val Loss: 0.5161, Train Acc: 0.9829, Val Acc: 0.7584
2026-07-07 23:34:32,985 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.0991, Val Loss: 0.5679, Train Acc: 0.9829, Val Acc: 0.6443
2026-07-07 23:34:33,073 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-07 23:34:46,934 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0986, Val Loss: 0.5931, Train Acc: 0.9829, Val Acc: 0.6040
2026-07-07 23:35:00,786 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.1018, Val Loss: 0.6658, Train Acc: 0.9829, Val Acc: 0.5503
2026-07-07 23:35:01,003 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 105, in train_normal_task
    test_loss, test_acc = tn.test_model(
                          ~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<5 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 87, in test_model
    outputs = model((mri, xls))
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 66, in forward
    image_out = self.image_section(image_data)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 36, in forward
    x = self.midflow(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 91, in forward
    a = nn.ELU()(self.block(x) + x)
                 ~~~~~~~~~~~~~~^~~
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:35:01,004 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)