2026-06-30 18:47:08,175 - ERROR - Pipeline failed: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 227, in run_background_pipeline
    task_func(self.update_progress, config)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
2026-06-30 18:47:12,955 - ERROR - Pipeline failed: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 227, in run_background_pipeline
    task_func(self.update_progress, config)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
2026-06-30 18:54:45,258 - ERROR - Pipeline failed: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 227, in run_background_pipeline
    task_func(self.update_progress, config)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
2026-06-30 19:08:20,961 - ERROR - Pipeline failed: App is not running
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 332, in run_background_pipeline
    task_func(self.update_progress, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 66, in dummy_task
    progress_callback(advance=1)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 381, in update_progress
    self.call_from_thread(_update)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/textual/app.py", line 1819, in call_from_thread
    raise RuntimeError("App is not running")
RuntimeError: App is not running
2026-06-30 19:18:14,215 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 19:22:46,181 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 21:49:24,173 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 21:58:23,232 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:02:17,566 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:02:24,454 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:05:28,118 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:06:52,271 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:07:09,800 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:07:16,120 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:07:25,317 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:08:39,971 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:08:44,617 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:09:01,476 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:09:52,766 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:14:38,011 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:15:31,941 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:19:35,748 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:25:58,294 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 22:26:28,903 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user while paused.
2026-06-30 22:37:54,542 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user while paused.
2026-06-30 22:38:05,493 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user while paused.
2026-06-30 23:00:47,768 - WARNING - Pipeline Interrupted: Pipeline execution stopped by user.
2026-06-30 23:09:29,204 - INFO - [SYSTEM] Application initialized and ready.
2026-06-30 23:13:16,084 - INFO - [SYSTEM] Application initialized and ready.
2026-06-30 23:13:17,733 - ERROR - [SYSTEM] No scenario selected! Please select a scenario from the dropdown.
2026-06-30 23:13:20,804 - INFO - [SYSTEM] Initiating Scenario: 2. Load, Evaluate, & Noise Analysis
2026-06-30 23:13:20,805 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:13:20,807 - INFO - [SYSTEM] Starting Task 1/3: Load Image and ADNIMERGE
2026-06-30 23:13:20,807 - ERROR - Pipeline failed: 'log'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 293, in run_background_pipeline
    task_func(task_tracker, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/tasks.py", line 10, in dummy_task
    log = state["log"]
          ~~~~~^^^^^^^
KeyError: 'log'
2026-06-30 23:13:20,808 - ERROR - [SYSTEM] ERROR: 'log'
2026-06-30 23:13:33,900 - INFO - [SYSTEM] Initiating Scenario: 2. Load, Evaluate, & Noise Analysis
2026-06-30 23:13:33,901 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:13:33,903 - INFO - [SYSTEM] Starting Task 1/3: Load Image and ADNIMERGE
2026-06-30 23:13:33,904 - ERROR - Pipeline failed: 'log'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 293, in run_background_pipeline
    task_func(task_tracker, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/tasks.py", line 10, in dummy_task
    log = state["log"]
          ~~~~~^^^^^^^
KeyError: 'log'
2026-06-30 23:13:33,904 - ERROR - [SYSTEM] ERROR: 'log'
2026-06-30 23:13:38,868 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-06-30 23:13:38,869 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:13:38,871 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-06-30 23:13:38,872 - ERROR - Pipeline failed: 'log'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 293, in run_background_pipeline
    task_func(task_tracker, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/tasks.py", line 10, in dummy_task
    log = state["log"]
          ~~~~~^^^^^^^
KeyError: 'log'
2026-06-30 23:13:38,872 - ERROR - [SYSTEM] ERROR: 'log'
2026-06-30 23:15:39,501 - INFO - [SYSTEM] Application initialized and ready.
2026-06-30 23:15:42,574 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-06-30 23:15:42,574 - INFO - [SYSTEM] Configuration saved to ./outputs/default/config.toml
2026-06-30 23:15:42,576 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-06-30 23:15:42,577 - INFO - [Load Image and ADNIMERGE] Initializing process...
2026-06-30 23:15:44,200 - INFO - [Load Image and ADNIMERGE] Halfway through current task execution...
2026-06-30 23:15:45,290 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-06-30 23:15:45,300 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-06-30 23:15:45,302 - INFO - [Train Regular Models] Initializing process...
2026-06-30 23:15:46,918 - INFO - [Train Regular Models] Halfway through current task execution...
2026-06-30 23:15:48,003 - INFO - [Train Regular Models] Task completed.
2026-06-30 23:15:48,004 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-06-30 23:15:48,005 - INFO - [Train Bayesian Models] Initializing process...
2026-06-30 23:15:49,636 - INFO - [Train Bayesian Models] Halfway through current task execution...
2026-06-30 23:15:50,716 - INFO - [Train Bayesian Models] Task completed.
2026-06-30 23:15:50,718 - INFO - [SYSTEM] Starting Task 4/5: Evaluate Regular Models
2026-06-30 23:15:50,719 - INFO - [Evaluate Regular Models] Initializing process...
2026-06-30 23:15:52,346 - INFO - [Evaluate Regular Models] Halfway through current task execution...
2026-06-30 23:15:53,414 - INFO - [Evaluate Regular Models] Task completed.
2026-06-30 23:15:53,416 - INFO - [SYSTEM] Starting Task 5/5: Evaluate Bayesian Models
2026-06-30 23:15:53,417 - INFO - [Evaluate Bayesian Models] Initializing process...
2026-06-30 23:15:53,671 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-06-30 23:15:53,685 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 20:54:06,351 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 20:54:46,133 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:01:54,340 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:03:07,022 - INFO - [SYSTEM] Copied default config to ./outputs/test1/config.toml. Ready for editing!
2026-07-07 21:04:00,870 - INFO - [SYSTEM] Successfully loaded config from ./outputs/test1/config.toml
2026-07-07 21:23:28,025 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:23:35,938 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 21:23:42,961 - ERROR - [SYSTEM] Invalid or missing scenario 'None' in config.toml!
2026-07-07 21:56:46,299 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:57:01,122 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 21:57:09,504 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 21:57:09,515 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 21:57:09,516 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 21:57:09,517 - ERROR - Pipeline failed: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 254, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 118, in load_adni_data_from_file
    xls_values = xls_preprocessor(pd.read_csv(xls_file))  # type: ignore
                                  ~~~~~~~~~~~^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 873, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 300, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1645, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1904, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/common.py", line 930, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 21:57:09,518 - ERROR - [SYSTEM] ERROR: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 21:57:20,610 - ERROR - [SYSTEM] Pipeline start cancelled by user (folder non-empty).
2026-07-07 21:58:55,263 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:59:21,987 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 21:59:23,885 - ERROR - [SYSTEM] Please specify a working directory.
2026-07-07 21:59:29,273 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 21:59:30,989 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 21:59:30,992 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 21:59:30,993 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 21:59:30,993 - ERROR - Pipeline failed: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 257, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 118, in load_adni_data_from_file
    xls_values = xls_preprocessor(pd.read_csv(xls_file))  # type: ignore
                                  ~~~~~~~~~~~^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 873, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 300, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1645, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/parsers/readers.py", line 1904, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/pandas/io/common.py", line 930, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 21:59:30,995 - ERROR - [SYSTEM] ERROR: [Errno 2] No such file or directory: 'LP_ADNIMERGE.csv'
2026-07-07 22:00:48,869 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:00:54,847 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:00:56,988 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:00:56,991 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:00:56,992 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:01:00,884 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:01:00,888 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:01:00,889 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:01:00,890 - INFO - [Train Regular Models] Initializing process...
2026-07-07 22:01:02,509 - INFO - [Train Regular Models] Halfway through current task execution...
2026-07-07 22:01:03,576 - INFO - [Train Regular Models] Task completed.
2026-07-07 22:01:03,577 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-07-07 22:01:03,578 - INFO - [Train Bayesian Models] Initializing process...
2026-07-07 22:01:04,289 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:01:04,321 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:40:31,759 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:40:40,591 - ERROR - [SYSTEM] Failed to initialize directory/config: [Errno 13] Permission denied: '/outputs'
2026-07-07 22:40:49,252 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-07 22:40:55,338 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:40:55,348 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:40:55,348 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:40:58,312 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:40:58,312 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:40:58,314 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:40:58,314 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:40:58,315 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:41:14,253 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 38.0169, Val Loss: 46.0747, Train Acc: 0.5034, Val Acc: 0.5034
2026-07-07 22:41:27,881 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 29.5272, Val Loss: 41.2137, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 22:41:30,945 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:41:41,559 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 25.7706, Val Loss: 35.2926, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 22:41:41,792 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 22:41:45,691 - INFO - [SYSTEM] Pipeline Resumed.
2026-07-07 22:41:50,686 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 22:41:55,302 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 24.3820, Val Loss: 38.6998, Train Acc: 0.5906, Val Acc: 0.5906
2026-07-07 22:42:09,063 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 21.6774, Val Loss: 39.5588, Train Acc: 0.6040, Val Acc: 0.6040
2026-07-07 22:42:22,836 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 20.0827, Val Loss: 41.8771, Train Acc: 0.5772, Val Acc: 0.5772
2026-07-07 22:42:36,641 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 18.2864, Val Loss: 37.1265, Train Acc: 0.5772, Val Acc: 0.5772
2026-07-07 22:42:36,744 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 22:42:50,611 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 18.5486, Val Loss: 39.2183, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 22:43:04,475 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 16.0196, Val Loss: 32.6778, Train Acc: 0.6577, Val Acc: 0.6577
2026-07-07 22:43:18,348 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 15.5705, Val Loss: 31.4642, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 22:43:32,211 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 14.0887, Val Loss: 45.1521, Train Acc: 0.5638, Val Acc: 0.5638
2026-07-07 22:45:59,606 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:46:09,907 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:46:11,087 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:46:11,099 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:46:11,100 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:46:13,818 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:46:13,823 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:46:13,824 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:46:13,824 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:46:13,825 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:46:13,826 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:46:28,132 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 35.8615, Val Loss: 39.1921, Train Acc: 0.5235, Val Acc: 0.5235
2026-07-07 22:46:36,897 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:46:37,458 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:47:11,214 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:47:11,217 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:47:11,217 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:47:13,919 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:47:13,920 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:47:13,925 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:47:13,925 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:47:13,926 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:47:13,927 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:47:27,633 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 31.7686, Val Loss: 35.9996, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 22:47:30,481 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:47:30,735 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:47:32,317 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:47:32,321 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:47:32,322 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:47:35,102 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:47:35,107 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:47:35,108 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:47:35,109 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:47:35,109 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:47:35,110 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:47:48,903 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 35.5566, Val Loss: 51.9955, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:48:02,651 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 27.3013, Val Loss: 55.1044, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:48:16,422 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 21.5175, Val Loss: 51.3943, Train Acc: 0.4832, Val Acc: 0.4832
2026-07-07 22:48:30,259 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 20.9605, Val Loss: 49.0898, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 22:48:44,110 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 18.0304, Val Loss: 46.6057, Train Acc: 0.4765, Val Acc: 0.4765
2026-07-07 22:48:57,963 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 16.3244, Val Loss: 49.9473, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:49:11,819 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 15.5546, Val Loss: 53.5298, Train Acc: 0.4631, Val Acc: 0.4631
2026-07-07 22:49:11,915 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 22:49:25,775 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 14.0157, Val Loss: 35.8735, Train Acc: 0.6107, Val Acc: 0.6107
2026-07-07 22:49:39,632 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 12.6665, Val Loss: 32.9891, Train Acc: 0.7181, Val Acc: 0.7181
2026-07-07 22:49:53,492 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 12.5009, Val Loss: 35.4068, Train Acc: 0.6107, Val Acc: 0.6107
2026-07-07 22:50:07,350 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 10.9609, Val Loss: 29.3653, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 22:50:21,206 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 11.1512, Val Loss: 38.1878, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 22:50:35,064 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 9.2618, Val Loss: 30.5188, Train Acc: 0.7315, Val Acc: 0.7315
2026-07-07 22:50:48,927 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 10.1587, Val Loss: 47.8058, Train Acc: 0.5235, Val Acc: 0.5235
2026-07-07 22:50:49,018 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 22:51:02,881 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 8.3822, Val Loss: 35.1154, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 22:51:16,742 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 9.1352, Val Loss: 36.4209, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 22:51:30,606 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 8.3755, Val Loss: 45.6383, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 22:51:44,471 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 7.2167, Val Loss: 30.6616, Train Acc: 0.7047, Val Acc: 0.7047
2026-07-07 22:51:58,333 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 7.5580, Val Loss: 34.0123, Train Acc: 0.6174, Val Acc: 0.6174
2026-07-07 22:52:12,203 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 6.0840, Val Loss: 64.6364, Train Acc: 0.4966, Val Acc: 0.4966
2026-07-07 22:52:12,253 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:52:26,061 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 8.6929, Val Loss: 40.2682, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 22:52:26,150 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 22:52:37,622 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:52:39,562 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:52:39,564 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:52:39,565 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:52:40,028 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 6.6049, Val Loss: 37.4700, Train Acc: 0.5906, Val Acc: 0.5906
2026-07-07 22:52:42,067 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 252, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:52:42,068 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:52:53,931 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 6.0106, Val Loss: 30.1979, Train Acc: 0.7315, Val Acc: 0.7315
2026-07-07 22:52:57,540 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:52:57,542 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:52:57,543 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:53:00,126 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 252, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:53:00,126 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:53:59,356 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:53:59,358 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:53:59,359 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:54:01,108 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 252, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:54:01,108 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 139412 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 22:56:14,146 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:56:27,820 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:56:31,532 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:56:31,535 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:56:31,535 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:56:34,443 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:56:34,447 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:56:34,448 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:56:34,449 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:56:34,449 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:56:34,450 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 22:56:48,915 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 31.9007, Val Loss: 40.0863, Train Acc: 0.5302, Val Acc: 0.5302
2026-07-07 22:56:55,874 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 22:56:55,915 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 22:59:41,802 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 22:59:51,565 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 22:59:53,442 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 22:59:53,445 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 22:59:53,445 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 22:59:56,393 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 22:59:56,397 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 22:59:56,398 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 22:59:56,399 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 22:59:56,399 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 22:59:56,400 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:00:10,718 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 30.2033, Val Loss: 40.4815, Train Acc: 0.5101, Val Acc: 0.5101
2026-07-07 23:00:16,121 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:00:16,157 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:00:19,835 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:00:19,836 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:00:19,837 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:00:22,440 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:00:22,444 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:00:22,445 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:00:22,446 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:00:22,446 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:00:22,447 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:00:27,800 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:00:27,946 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:00:56,651 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:00:56,653 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:00:56,654 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:00:58,796 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:00:58,800 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:00:58,801 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:00:58,801 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:00:58,802 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:00:58,802 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:01:12,503 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 33.8031, Val Loss: 38.0120, Train Acc: 0.7919, Val Acc: 0.7919
2026-07-07 23:01:26,219 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 27.8724, Val Loss: 40.8105, Train Acc: 0.4966, Val Acc: 0.4966
2026-07-07 23:01:39,984 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 25.4381, Val Loss: 35.0456, Train Acc: 0.6644, Val Acc: 0.6644
2026-07-07 23:01:53,781 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 23.3444, Val Loss: 35.3899, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 23:02:07,612 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 21.0827, Val Loss: 33.5305, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 23:02:21,436 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 20.1785, Val Loss: 37.8223, Train Acc: 0.7517, Val Acc: 0.7517
2026-07-07 23:02:35,268 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 18.5837, Val Loss: 43.5106, Train Acc: 0.5168, Val Acc: 0.5168
2026-07-07 23:02:35,344 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:02:38,077 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:02:38,490 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:05:49,105 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:06:05,829 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-07 23:06:08,440 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:06:08,451 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:06:08,451 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:06:11,353 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:06:11,357 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:06:11,358 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:06:11,359 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:06:11,359 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:06:11,360 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:06:19,433 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:06:19,831 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:07:23,454 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:07:23,457 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:07:23,457 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:07:26,158 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:07:26,162 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:07:26,163 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:07:26,164 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:07:26,164 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:07:26,165 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:07:39,894 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 36.3690, Val Loss: 39.1494, Train Acc: 0.6242, Val Acc: 0.6242
2026-07-07 23:07:47,496 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 23:07:49,550 - INFO - [SYSTEM] Pipeline Resumed.
2026-07-07 23:07:52,492 - INFO - [SYSTEM] Pipeline Paused. Waiting for current operation to yield...
2026-07-07 23:07:54,747 - INFO - [SYSTEM] Pipeline Resumed.
2026-07-07 23:07:57,610 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 28.3761, Val Loss: 42.0669, Train Acc: 0.5034, Val Acc: 0.5034
2026-07-07 23:07:59,573 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:07:59,949 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:08:44,258 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:08:55,979 - INFO - [SYSTEM] Copied default config to ./outputs/exp2/config.toml. Ready for editing!
2026-07-07 23:09:00,092 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:09:00,094 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:09:00,095 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:09:02,563 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:02,564 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:11,258 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:09:11,272 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:09:11,273 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:09:13,452 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/load_data.py", line 22, in load_data_task
    dataset = ds.load_adni_data_from_file(
        mri_files,
    ...<2 lines>...
        xls_preprocessor=ds.xls_pre,
    )
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 177, in load_adni_data_from_file
    return ADNIDataset(mri_data, xls_data, expected_classes, img_ids, device=device)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dataset.py", line 62, in __init__
    self.mri_data = mri_data.float().to(device)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:13,453 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 2.60 GiB. GPU 0 has a total capacity of 11.63 GiB of which 111.31 MiB is free. Process 143180 has 9.47 GiB memory in use. Including non-PyTorch memory, this process has 250.00 MiB memory in use. Of the allocated memory 0 bytes is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:09:50,869 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:09:58,158 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp2/config.toml
2026-07-07 23:09:59,995 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:09:59,998 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:09:59,998 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:10:02,945 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:10:02,950 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:10:02,951 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:10:02,952 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:10:02,952 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp2/normal_models/intermediate_models
2026-07-07 23:10:02,953 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:10:17,381 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 34.3198, Val Loss: 37.7880, Train Acc: 0.7450, Val Acc: 0.7450
2026-07-07 23:10:31,134 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 28.2604, Val Loss: 35.1542, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 23:10:44,929 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 25.2059, Val Loss: 37.1065, Train Acc: 0.6443, Val Acc: 0.6443
2026-07-07 23:10:58,741 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 23.2649, Val Loss: 37.7584, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 23:11:12,573 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 20.9050, Val Loss: 36.4091, Train Acc: 0.6040, Val Acc: 0.6040
2026-07-07 23:11:26,413 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 19.2700, Val Loss: 36.3774, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 23:11:40,255 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 18.0514, Val Loss: 35.6015, Train Acc: 0.6376, Val Acc: 0.6376
2026-07-07 23:11:40,355 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:11:54,199 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 17.0273, Val Loss: 34.6692, Train Acc: 0.6309, Val Acc: 0.6309
2026-07-07 23:12:08,042 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 16.3381, Val Loss: 56.3836, Train Acc: 0.5369, Val Acc: 0.5369
2026-07-07 23:12:21,887 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 17.3398, Val Loss: 38.8979, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 23:12:35,737 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 14.4686, Val Loss: 30.7893, Train Acc: 0.8054, Val Acc: 0.8054
2026-07-07 23:12:49,582 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 13.8248, Val Loss: 35.9297, Train Acc: 0.6242, Val Acc: 0.6242
2026-07-07 23:13:03,427 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 13.3095, Val Loss: 35.1133, Train Acc: 0.7181, Val Acc: 0.7181
2026-07-07 23:13:17,272 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 12.7092, Val Loss: 44.7456, Train Acc: 0.5638, Val Acc: 0.5638
2026-07-07 23:13:17,359 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 23:13:31,211 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 13.0241, Val Loss: 49.0018, Train Acc: 0.5503, Val Acc: 0.5503
2026-07-07 23:13:45,062 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 11.3344, Val Loss: 40.5898, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 23:13:58,917 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 10.7968, Val Loss: 33.3214, Train Acc: 0.6577, Val Acc: 0.6577
2026-07-07 23:14:12,766 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 9.2847, Val Loss: 32.2253, Train Acc: 0.7919, Val Acc: 0.7919
2026-07-07 23:14:26,619 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 9.5039, Val Loss: 28.7371, Train Acc: 0.8523, Val Acc: 0.8523
2026-07-07 23:14:40,468 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 8.9888, Val Loss: 35.7847, Train Acc: 0.6107, Val Acc: 0.6107
2026-07-07 23:14:54,317 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 9.0884, Val Loss: 32.0707, Train Acc: 0.6644, Val Acc: 0.6644
2026-07-07 23:14:54,411 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 23:15:08,264 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 8.4898, Val Loss: 38.5546, Train Acc: 0.5839, Val Acc: 0.5839
2026-07-07 23:15:22,114 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 7.8137, Val Loss: 33.0331, Train Acc: 0.7114, Val Acc: 0.7114
2026-07-07 23:15:35,966 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 7.0856, Val Loss: 27.0260, Train Acc: 0.8054, Val Acc: 0.8054
2026-07-07 23:15:49,811 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 7.6837, Val Loss: 45.5512, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 23:16:03,659 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 6.6017, Val Loss: 32.3892, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 23:16:17,508 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 7.7363, Val Loss: 34.3906, Train Acc: 0.7852, Val Acc: 0.7852
2026-07-07 23:16:31,357 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 7.1077, Val Loss: 41.1836, Train Acc: 0.5705, Val Acc: 0.5705
2026-07-07 23:16:31,445 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-07 23:16:45,297 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 6.1704, Val Loss: 38.0425, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 23:16:59,145 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 6.1988, Val Loss: 29.8566, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 23:16:59,362 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 96, in train_normal_task
    test_loss, test_acc = tn.test_model(
                          ~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<5 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 65, in test_model
    outputs = model((mri, xls))
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 66, in forward
    image_out = self.image_section(image_data)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 36, in forward
    x = self.midflow(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 91, in forward
    a = nn.ELU()(self.block(x) + x)
                 ~~~~~~~~~~~~~~^~~
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:16:59,370 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:18:31,229 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:18:31,236 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:18:31,237 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:18:34,007 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:18:34,009 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:18:34,010 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:18:34,011 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:18:34,011 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp2/normal_models/intermediate_models
2026-07-07 23:18:34,012 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:18:39,480 - ERROR - Pipeline failed: App is not running
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 81, in train_normal_task
    model, history = tn.train_model(
                     ~~~~~~~~~~~~~~^
        log=log,
        ^^^^^^^^
    ...<9 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 190, in train_model
    train_loss, val_loss, train_acc, val_acc = train_epoch(
                                               ~~~~~~~~~~~^
        model,
        ^^^^^^
    ...<7 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 120, in train_epoch
    train_progress.update(total=len(train_loader), advance=1)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/progress.py", line 64, in update
    self.app.call_from_thread(_do_update)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/textual/app.py", line 1819, in call_from_thread
    raise RuntimeError("App is not running")
RuntimeError: App is not running
2026-07-07 23:18:39,481 - ERROR - [SYSTEM] ERROR: App is not running
2026-07-07 23:22:18,712 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:22:30,814 - INFO - [SYSTEM] Copied default config to ./outputs/experiment_1/config.toml. Ready for editing!
2026-07-07 23:22:33,000 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:22:33,003 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:22:33,004 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:22:35,714 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:22:35,718 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:22:35,720 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:22:35,720 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:22:35,721 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/experiment_1/normal_models/intermediate_models
2026-07-07 23:22:35,721 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:22:50,086 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 37.1905, Val Loss: 42.7157, Train Acc: 0.5034, Val Acc: 0.5034
2026-07-07 23:23:03,790 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 25.8338, Val Loss: 35.7036, Train Acc: 0.7383, Val Acc: 0.7383
2026-07-07 23:23:17,550 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 21.3707, Val Loss: 45.1811, Train Acc: 0.4765, Val Acc: 0.4765
2026-07-07 23:23:31,351 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 19.2528, Val Loss: 36.2448, Train Acc: 0.5973, Val Acc: 0.5973
2026-07-07 23:23:45,172 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 16.2027, Val Loss: 32.4055, Train Acc: 0.6913, Val Acc: 0.6913
2026-07-07 23:23:58,997 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 14.6334, Val Loss: 30.1882, Train Acc: 0.8255, Val Acc: 0.8255
2026-07-07 23:24:06,489 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-07 23:24:06,847 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-07 23:27:52,571 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:27:59,544 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-07 23:28:01,690 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:28:01,701 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:28:01,701 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:28:04,661 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:28:04,663 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:28:04,668 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:28:04,668 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:28:04,669 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-07 23:28:04,669 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:28:18,992 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5685, Val Loss: 0.7230, Train Acc: 0.7034, Val Acc: 0.4631
2026-07-07 23:28:32,736 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4527, Val Loss: 0.7075, Train Acc: 0.8365, Val Acc: 0.4966
2026-07-07 23:28:46,493 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.3957, Val Loss: 0.5646, Train Acc: 0.9125, Val Acc: 0.8188
2026-07-07 23:29:00,299 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3627, Val Loss: 0.7107, Train Acc: 0.9506, Val Acc: 0.5101
2026-07-07 23:29:14,125 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3393, Val Loss: 0.6061, Train Acc: 0.9715, Val Acc: 0.7181
2026-07-07 23:29:27,967 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.3099, Val Loss: 0.6734, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-07 23:29:41,808 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2903, Val Loss: 0.6156, Train Acc: 0.9810, Val Acc: 0.5973
2026-07-07 23:29:41,901 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:29:55,752 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2748, Val Loss: 0.6656, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-07 23:30:09,598 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2596, Val Loss: 0.5948, Train Acc: 0.9848, Val Acc: 0.6242
2026-07-07 23:30:23,445 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2503, Val Loss: 0.5690, Train Acc: 0.9791, Val Acc: 0.7987
2026-07-07 23:30:37,293 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.2408, Val Loss: 0.8184, Train Acc: 0.9829, Val Acc: 0.5235
2026-07-07 23:30:51,140 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.2546, Val Loss: 0.6428, Train Acc: 0.9696, Val Acc: 0.5906
2026-07-07 23:31:04,990 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.2260, Val Loss: 0.6634, Train Acc: 0.9848, Val Acc: 0.5839
2026-07-07 23:31:18,836 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.2089, Val Loss: 0.6569, Train Acc: 0.9829, Val Acc: 0.5772
2026-07-07 23:31:18,924 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 23:31:32,776 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1820, Val Loss: 0.5285, Train Acc: 0.9924, Val Acc: 0.7651
2026-07-07 23:31:46,627 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1814, Val Loss: 0.5080, Train Acc: 0.9848, Val Acc: 0.8054
2026-07-07 23:32:00,481 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1848, Val Loss: 0.5403, Train Acc: 0.9810, Val Acc: 0.8255
2026-07-07 23:32:14,331 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1725, Val Loss: 0.5621, Train Acc: 0.9867, Val Acc: 0.6577
2026-07-07 23:32:28,183 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1524, Val Loss: 0.8701, Train Acc: 0.9848, Val Acc: 0.5034
2026-07-07 23:32:42,035 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1537, Val Loss: 0.6627, Train Acc: 0.9848, Val Acc: 0.5906
2026-07-07 23:32:55,889 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1386, Val Loss: 0.5572, Train Acc: 0.9829, Val Acc: 0.6510
2026-07-07 23:32:55,982 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 23:33:09,838 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1314, Val Loss: 0.5282, Train Acc: 0.9867, Val Acc: 0.7919
2026-07-07 23:33:23,693 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.1305, Val Loss: 0.4771, Train Acc: 0.9848, Val Acc: 0.8523
2026-07-07 23:33:37,549 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1217, Val Loss: 0.7991, Train Acc: 0.9848, Val Acc: 0.5436
2026-07-07 23:33:51,407 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.1192, Val Loss: 0.6912, Train Acc: 0.9867, Val Acc: 0.5503
2026-07-07 23:34:05,268 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.1029, Val Loss: 0.6175, Train Acc: 0.9886, Val Acc: 0.6443
2026-07-07 23:34:19,129 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.1157, Val Loss: 0.5161, Train Acc: 0.9829, Val Acc: 0.7584
2026-07-07 23:34:32,985 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.0991, Val Loss: 0.5679, Train Acc: 0.9829, Val Acc: 0.6443
2026-07-07 23:34:33,073 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-07 23:34:46,934 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0986, Val Loss: 0.5931, Train Acc: 0.9829, Val Acc: 0.6040
2026-07-07 23:35:00,786 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.1018, Val Loss: 0.6658, Train Acc: 0.9829, Val Acc: 0.5503
2026-07-07 23:35:01,003 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 253, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 105, in train_normal_task
    test_loss, test_acc = tn.test_model(
                          ~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<5 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 87, in test_model
    outputs = model((mri, xls))
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 66, in forward
    image_out = self.image_section(image_data)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 36, in forward
    x = self.midflow(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 91, in forward
    a = nn.ELU()(self.block(x) + x)
                 ~~~~~~~~~~~~~~^~~
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:35:01,004 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 272.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 94.62 MiB is free. Including non-PyTorch memory, this process has 9.73 GiB memory in use. Of the allocated memory 8.46 GiB is allocated by PyTorch, and 990.38 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-07 23:42:26,276 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:42:34,766 - INFO - [SYSTEM] Copied default config to ./outputs/exp3/config.toml. Ready for editing!
2026-07-07 23:42:35,810 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:42:35,878 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:42:35,878 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:42:38,392 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:42:38,397 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:42:38,398 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:42:38,398 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:42:38,399 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp3/normal_models/intermediate_models
2026-07-07 23:42:38,400 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:42:38,490 - ERROR - Pipeline failed: Expected 'True' or 'False' at index 2 in ConfigTokenizer but got 'true'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 256, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 79, in train_normal_task
    .to(config["training"]["device"])
     ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1383, in to
    return self._apply(convert)
           ~~~~~~~~~~~^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
    ~~~~~~~~~~~~~^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
    ~~~~~~~~~~~~~^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
    ~~~~~~~~~~~~~^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 964, in _apply
    param_applied = fn(param)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1369, in convert
    return t.to(
           ~~~~^
        device,
        ^^^^^^^
        dtype if t.is_floating_point() or t.is_complex() else None,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        non_blocking,
        ^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/cuda/__init__.py", line 491, in _lazy_init
    torch._C._cuda_init()
    ~~~~~~~~~~~~~~~~~~~^^
ValueError: Expected 'True' or 'False' at index 2 in ConfigTokenizer but got 'true'
2026-07-07 23:42:38,491 - ERROR - [SYSTEM] ERROR: Expected 'True' or 'False' at index 2 in ConfigTokenizer but got 'true'
2026-07-07 23:44:33,294 - INFO - [SYSTEM] Application initialized and ready.
2026-07-07 23:44:42,536 - INFO - [SYSTEM] Copied default config to ./outputs/exp2/config.toml. Ready for editing!
2026-07-07 23:44:44,836 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-07 23:44:44,839 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-07 23:44:44,840 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-07 23:44:47,429 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-07 23:44:47,434 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-07 23:44:47,435 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-07 23:44:47,436 - INFO - [Train Regular Models] Starting normal model training...
2026-07-07 23:44:47,436 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp2/normal_models/intermediate_models
2026-07-07 23:44:47,437 - INFO - [Train Regular Models] Training model 1/50...
2026-07-07 23:45:02,439 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5988, Val Loss: 0.6774, Train Acc: 0.6863, Val Acc: 0.5570
2026-07-07 23:45:16,620 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4517, Val Loss: 0.6813, Train Acc: 0.8631, Val Acc: 0.4966
2026-07-07 23:45:30,845 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.3659, Val Loss: 0.7556, Train Acc: 0.9202, Val Acc: 0.4832
2026-07-07 23:45:45,139 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3124, Val Loss: 0.6634, Train Acc: 0.9715, Val Acc: 0.4899
2026-07-07 23:45:59,466 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.2674, Val Loss: 0.5152, Train Acc: 0.9734, Val Acc: 0.7651
2026-07-07 23:46:13,794 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.2362, Val Loss: 0.5708, Train Acc: 0.9810, Val Acc: 0.7987
2026-07-07 23:46:28,149 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2227, Val Loss: 0.7055, Train Acc: 0.9829, Val Acc: 0.5503
2026-07-07 23:46:28,246 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:46:42,586 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2107, Val Loss: 0.8692, Train Acc: 0.9734, Val Acc: 0.5168
2026-07-07 23:46:56,924 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2310, Val Loss: 0.6027, Train Acc: 0.9715, Val Acc: 0.5772
2026-07-07 23:47:11,269 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.1790, Val Loss: 0.5413, Train Acc: 0.9810, Val Acc: 0.6846
2026-07-07 23:47:25,608 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.1858, Val Loss: 0.5737, Train Acc: 0.9753, Val Acc: 0.8121
2026-07-07 23:47:39,962 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.1628, Val Loss: 0.6267, Train Acc: 0.9791, Val Acc: 0.5772
2026-07-07 23:47:54,321 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.1556, Val Loss: 0.6652, Train Acc: 0.9772, Val Acc: 0.6174
2026-07-07 23:48:08,673 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.1475, Val Loss: 0.6061, Train Acc: 0.9848, Val Acc: 0.6309
2026-07-07 23:48:08,761 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 23:48:23,106 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1268, Val Loss: 0.6765, Train Acc: 0.9829, Val Acc: 0.5235
2026-07-07 23:48:37,459 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1323, Val Loss: 0.4715, Train Acc: 0.9829, Val Acc: 0.7987
2026-07-07 23:48:51,811 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1211, Val Loss: 0.4280, Train Acc: 0.9810, Val Acc: 0.8121
2026-07-07 23:49:06,167 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1296, Val Loss: 0.6625, Train Acc: 0.9791, Val Acc: 0.5503
2026-07-07 23:49:20,516 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1093, Val Loss: 0.4699, Train Acc: 0.9829, Val Acc: 0.7517
2026-07-07 23:49:34,971 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.0989, Val Loss: 0.5191, Train Acc: 0.9848, Val Acc: 0.7181
2026-07-07 23:49:49,308 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1026, Val Loss: 0.6802, Train Acc: 0.9886, Val Acc: 0.5839
2026-07-07 23:49:49,402 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 23:50:03,731 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.0917, Val Loss: 0.5392, Train Acc: 0.9829, Val Acc: 0.6644
2026-07-07 23:50:18,069 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.0925, Val Loss: 0.4311, Train Acc: 0.9791, Val Acc: 0.8188
2026-07-07 23:50:32,409 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.0852, Val Loss: 0.4381, Train Acc: 0.9810, Val Acc: 0.8389
2026-07-07 23:50:46,754 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.0855, Val Loss: 0.5525, Train Acc: 0.9867, Val Acc: 0.6242
2026-07-07 23:51:01,110 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.0784, Val Loss: 0.5054, Train Acc: 0.9848, Val Acc: 0.7785
2026-07-07 23:51:15,446 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.0831, Val Loss: 0.5432, Train Acc: 0.9848, Val Acc: 0.6443
2026-07-07 23:51:29,777 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.0686, Val Loss: 0.4149, Train Acc: 0.9886, Val Acc: 0.8389
2026-07-07 23:51:29,866 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-07 23:51:44,208 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0676, Val Loss: 0.3909, Train Acc: 0.9810, Val Acc: 0.8523
2026-07-07 23:51:58,561 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.0719, Val Loss: 0.8435, Train Acc: 0.9848, Val Acc: 0.5369
2026-07-07 23:51:59,261 - INFO - [Train Regular Models] Run 1/50 - Test Loss: 0.6056, Test Accuracy: 0.6495
2026-07-07 23:51:59,364 - INFO - [Train Regular Models] Model saved to outputs/exp2/normal_models/model_1.pt
2026-07-07 23:51:59,633 - INFO - [Train Regular Models] Training model 2/50...
2026-07-07 23:52:14,066 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.6460, Val Loss: 0.7060, Train Acc: 0.6445, Val Acc: 0.4899
2026-07-07 23:52:28,410 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4908, Val Loss: 0.6186, Train Acc: 0.8384, Val Acc: 0.7181
2026-07-07 23:52:42,760 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.4295, Val Loss: 0.5950, Train Acc: 0.9144, Val Acc: 0.6913
2026-07-07 23:52:57,106 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3802, Val Loss: 0.6587, Train Acc: 0.9639, Val Acc: 0.5638
2026-07-07 23:53:11,456 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3412, Val Loss: 0.6661, Train Acc: 0.9696, Val Acc: 0.5772
2026-07-07 23:53:25,804 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.3171, Val Loss: 0.6438, Train Acc: 0.9734, Val Acc: 0.5839
2026-07-07 23:53:40,156 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2895, Val Loss: 0.5496, Train Acc: 0.9734, Val Acc: 0.6846
2026-07-07 23:53:40,294 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-07 23:53:54,646 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2662, Val Loss: 0.5259, Train Acc: 0.9772, Val Acc: 0.8255
2026-07-07 23:54:09,010 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2498, Val Loss: 0.6759, Train Acc: 0.9829, Val Acc: 0.5772
2026-07-07 23:54:23,368 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2406, Val Loss: 0.4936, Train Acc: 0.9734, Val Acc: 0.8523
2026-07-07 23:54:37,708 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.2321, Val Loss: 0.5880, Train Acc: 0.9753, Val Acc: 0.6577
2026-07-07 23:54:52,060 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.2239, Val Loss: 0.7252, Train Acc: 0.9753, Val Acc: 0.5638
2026-07-07 23:55:06,420 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.2010, Val Loss: 0.7599, Train Acc: 0.9829, Val Acc: 0.5436
2026-07-07 23:55:20,750 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.1926, Val Loss: 0.5811, Train Acc: 0.9753, Val Acc: 0.6309
2026-07-07 23:55:20,881 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-07 23:55:35,238 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1724, Val Loss: 0.6586, Train Acc: 0.9848, Val Acc: 0.5638
2026-07-07 23:55:49,589 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1870, Val Loss: 0.6229, Train Acc: 0.9734, Val Acc: 0.5772
2026-07-07 23:56:03,939 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1716, Val Loss: 0.5946, Train Acc: 0.9734, Val Acc: 0.6107
2026-07-07 23:56:18,289 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1540, Val Loss: 0.6836, Train Acc: 0.9810, Val Acc: 0.5772
2026-07-07 23:56:32,641 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1469, Val Loss: 0.8042, Train Acc: 0.9791, Val Acc: 0.5436
2026-07-07 23:56:46,998 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1332, Val Loss: 0.5873, Train Acc: 0.9886, Val Acc: 0.6376
2026-07-07 23:57:01,335 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1288, Val Loss: 0.4624, Train Acc: 0.9848, Val Acc: 0.7718
2026-07-07 23:57:01,467 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-07 23:57:15,812 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1168, Val Loss: 0.5633, Train Acc: 0.9829, Val Acc: 0.6711
2026-07-07 23:57:30,163 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.1279, Val Loss: 0.4908, Train Acc: 0.9829, Val Acc: 0.8188
2026-07-07 23:57:44,504 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1189, Val Loss: 0.5049, Train Acc: 0.9867, Val Acc: 0.7450
2026-07-07 23:57:58,843 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.1111, Val Loss: 0.5347, Train Acc: 0.9829, Val Acc: 0.6980
2026-07-07 23:58:13,193 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.0969, Val Loss: 0.6163, Train Acc: 0.9867, Val Acc: 0.6107
2026-07-07 23:58:27,554 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.1007, Val Loss: 0.4294, Train Acc: 0.9829, Val Acc: 0.8188
2026-07-07 23:58:41,904 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.0938, Val Loss: 0.6080, Train Acc: 0.9848, Val Acc: 0.6040
2026-07-07 23:58:42,034 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-07 23:58:56,375 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0919, Val Loss: 0.5183, Train Acc: 0.9867, Val Acc: 0.6980
2026-07-07 23:59:10,718 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.0870, Val Loss: 0.5387, Train Acc: 0.9886, Val Acc: 0.6644
2026-07-07 23:59:11,395 - INFO - [Train Regular Models] Run 2/50 - Test Loss: 0.4164, Test Accuracy: 0.7938
2026-07-07 23:59:11,483 - INFO - [Train Regular Models] Model saved to outputs/exp2/normal_models/model_2.pt
2026-07-07 23:59:11,761 - INFO - [Train Regular Models] Training model 3/50...
2026-07-07 23:59:26,211 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5464, Val Loss: 0.6425, Train Acc: 0.8061, Val Acc: 0.7114
2026-07-07 23:59:40,558 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4258, Val Loss: 0.5892, Train Acc: 0.9316, Val Acc: 0.7919
2026-07-07 23:59:54,918 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.3906, Val Loss: 0.5350, Train Acc: 0.9487, Val Acc: 0.8456
2026-07-08 00:00:09,260 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3363, Val Loss: 0.5485, Train Acc: 0.9715, Val Acc: 0.7181
2026-07-08 00:00:23,605 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3088, Val Loss: 0.5170, Train Acc: 0.9715, Val Acc: 0.8456
2026-07-08 00:00:37,960 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.2832, Val Loss: 0.5551, Train Acc: 0.9696, Val Acc: 0.8188
2026-07-08 00:00:52,324 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2756, Val Loss: 0.6875, Train Acc: 0.9677, Val Acc: 0.5503
2026-07-08 00:00:52,461 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-08 00:01:06,804 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2526, Val Loss: 0.6073, Train Acc: 0.9715, Val Acc: 0.6309
2026-07-08 00:01:21,153 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2352, Val Loss: 0.5923, Train Acc: 0.9772, Val Acc: 0.5906
2026-07-08 00:01:35,493 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2159, Val Loss: 0.5506, Train Acc: 0.9791, Val Acc: 0.7114
2026-07-08 00:01:49,840 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.2014, Val Loss: 0.5502, Train Acc: 0.9772, Val Acc: 0.6644
2026-07-08 00:02:04,204 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.1865, Val Loss: 0.6347, Train Acc: 0.9886, Val Acc: 0.5973
2026-07-08 00:02:18,557 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.1671, Val Loss: 0.4521, Train Acc: 0.9753, Val Acc: 0.8389
2026-07-08 00:02:32,920 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.1858, Val Loss: 0.5306, Train Acc: 0.9658, Val Acc: 0.8188
2026-07-08 00:02:33,052 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-08 00:02:47,405 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1645, Val Loss: 0.5667, Train Acc: 0.9810, Val Acc: 0.7047
2026-07-08 00:03:01,749 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1612, Val Loss: 0.6132, Train Acc: 0.9867, Val Acc: 0.5772
2026-07-08 00:03:16,204 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1429, Val Loss: 0.6235, Train Acc: 0.9829, Val Acc: 0.6711
2026-07-08 00:03:30,552 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1414, Val Loss: 0.5035, Train Acc: 0.9867, Val Acc: 0.8389
2026-07-08 00:03:44,890 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1314, Val Loss: 0.6260, Train Acc: 0.9848, Val Acc: 0.5973
2026-07-08 00:03:59,240 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1230, Val Loss: 0.5351, Train Acc: 0.9810, Val Acc: 0.7383
2026-07-08 00:04:13,601 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1154, Val Loss: 0.5500, Train Acc: 0.9848, Val Acc: 0.6577
2026-07-08 00:04:13,734 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-08 00:04:28,078 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1065, Val Loss: 0.4489, Train Acc: 0.9848, Val Acc: 0.8255
2026-07-08 00:04:42,434 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.1042, Val Loss: 0.4787, Train Acc: 0.9791, Val Acc: 0.7584
2026-07-08 00:04:56,788 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1047, Val Loss: 0.6119, Train Acc: 0.9791, Val Acc: 0.5973
2026-07-08 00:05:11,140 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.0935, Val Loss: 0.4899, Train Acc: 0.9867, Val Acc: 0.8054
2026-07-08 00:05:25,504 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.1009, Val Loss: 0.4722, Train Acc: 0.9848, Val Acc: 0.7919
2026-07-08 00:05:39,844 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.0825, Val Loss: 0.7348, Train Acc: 0.9848, Val Acc: 0.5638
2026-07-08 00:05:40,750 - ERROR - [SYSTEM] Stop requested! Terminating gracefully...
2026-07-08 00:05:41,494 - ERROR - [SYSTEM] STOPPED: Pipeline execution stopped by user.
2026-07-08 00:06:24,491 - INFO - [SYSTEM] Application initialized and ready.
2026-07-08 00:06:34,726 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:06:38,151 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-08 00:06:38,154 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-08 00:06:38,155 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-08 00:06:40,730 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-08 00:06:40,735 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-08 00:06:40,736 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-08 00:06:40,737 - INFO - [Train Regular Models] Starting normal model training...
2026-07-08 00:06:40,737 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-08 00:06:40,738 - INFO - [Train Regular Models] Training model 1/3...
2026-07-08 00:06:55,866 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5781, Val Loss: 0.6663, Train Acc: 0.7510, Val Acc: 0.6242
2026-07-08 00:07:10,105 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4435, Val Loss: 0.5940, Train Acc: 0.8859, Val Acc: 0.7248
2026-07-08 00:07:24,388 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.3795, Val Loss: 0.5657, Train Acc: 0.9411, Val Acc: 0.6846
2026-07-08 00:07:38,704 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3532, Val Loss: 0.5478, Train Acc: 0.9544, Val Acc: 0.6980
2026-07-08 00:07:53,087 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3132, Val Loss: 0.6986, Train Acc: 0.9658, Val Acc: 0.5503
2026-07-08 00:08:07,471 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.3196, Val Loss: 0.6372, Train Acc: 0.9620, Val Acc: 0.5839
2026-07-08 00:08:21,870 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2833, Val Loss: 0.7402, Train Acc: 0.9715, Val Acc: 0.5503
2026-07-08 00:08:21,970 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-08 00:08:36,364 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2652, Val Loss: 0.6051, Train Acc: 0.9753, Val Acc: 0.5973
2026-07-08 00:08:50,759 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2387, Val Loss: 0.5096, Train Acc: 0.9715, Val Acc: 0.7315
2026-07-08 00:09:05,163 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2328, Val Loss: 0.5193, Train Acc: 0.9810, Val Acc: 0.7315
2026-07-08 00:09:19,552 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.2022, Val Loss: 0.4929, Train Acc: 0.9810, Val Acc: 0.8591
2026-07-08 00:09:33,940 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.1841, Val Loss: 0.4870, Train Acc: 0.9867, Val Acc: 0.7919
2026-07-08 00:09:48,337 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.1790, Val Loss: 0.5048, Train Acc: 0.9791, Val Acc: 0.6980
2026-07-08 00:10:02,734 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.1669, Val Loss: 0.5807, Train Acc: 0.9829, Val Acc: 0.6107
2026-07-08 00:10:02,824 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-08 00:10:17,214 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1697, Val Loss: 0.9612, Train Acc: 0.9810, Val Acc: 0.4899
2026-07-08 00:10:31,609 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1613, Val Loss: 0.5998, Train Acc: 0.9810, Val Acc: 0.5772
2026-07-08 00:10:45,982 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1501, Val Loss: 0.5639, Train Acc: 0.9734, Val Acc: 0.7047
2026-07-08 00:11:00,372 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1421, Val Loss: 0.5669, Train Acc: 0.9791, Val Acc: 0.7450
2026-07-08 00:11:14,767 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1332, Val Loss: 0.5990, Train Acc: 0.9810, Val Acc: 0.7315
2026-07-08 00:11:29,168 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1245, Val Loss: 0.4664, Train Acc: 0.9829, Val Acc: 0.8658
2026-07-08 00:11:43,564 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1183, Val Loss: 0.5506, Train Acc: 0.9829, Val Acc: 0.6510
2026-07-08 00:11:43,656 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-08 00:11:58,039 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1113, Val Loss: 0.5285, Train Acc: 0.9829, Val Acc: 0.7919
2026-07-08 00:12:12,517 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.1080, Val Loss: 0.6438, Train Acc: 0.9810, Val Acc: 0.5906
2026-07-08 00:12:26,904 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1057, Val Loss: 0.5118, Train Acc: 0.9848, Val Acc: 0.7383
2026-07-08 00:12:41,286 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.0973, Val Loss: 0.5503, Train Acc: 0.9848, Val Acc: 0.6779
2026-07-08 00:12:55,651 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.1044, Val Loss: 0.5491, Train Acc: 0.9829, Val Acc: 0.7785
2026-07-08 00:13:10,036 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.0996, Val Loss: 0.5606, Train Acc: 0.9829, Val Acc: 0.6980
2026-07-08 00:13:24,420 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.0843, Val Loss: 0.4561, Train Acc: 0.9867, Val Acc: 0.7785
2026-07-08 00:13:24,509 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-08 00:13:38,894 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0812, Val Loss: 0.5308, Train Acc: 0.9886, Val Acc: 0.7785
2026-07-08 00:13:53,281 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.0887, Val Loss: 0.4591, Train Acc: 0.9791, Val Acc: 0.8322
2026-07-08 00:13:53,973 - INFO - [Train Regular Models] Run 1/3 - Test Loss: 0.3966, Test Accuracy: 0.8763
2026-07-08 00:13:54,078 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_1.pt
2026-07-08 00:13:54,345 - INFO - [Train Regular Models] Training model 2/3...
2026-07-08 00:14:08,824 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5572, Val Loss: 0.6509, Train Acc: 0.7814, Val Acc: 0.5503
2026-07-08 00:14:23,196 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4244, Val Loss: 0.5700, Train Acc: 0.8916, Val Acc: 0.7584
2026-07-08 00:14:37,576 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.3840, Val Loss: 0.6195, Train Acc: 0.9221, Val Acc: 0.6107
2026-07-08 00:14:51,953 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3291, Val Loss: 0.5749, Train Acc: 0.9620, Val Acc: 0.6309
2026-07-08 00:15:06,336 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3090, Val Loss: 0.5754, Train Acc: 0.9753, Val Acc: 0.6711
2026-07-08 00:15:20,719 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.2822, Val Loss: 0.5733, Train Acc: 0.9715, Val Acc: 0.6107
2026-07-08 00:15:35,114 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2556, Val Loss: 0.5580, Train Acc: 0.9772, Val Acc: 0.6913
2026-07-08 00:15:35,251 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-08 00:15:49,621 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2384, Val Loss: 0.4849, Train Acc: 0.9753, Val Acc: 0.8322
2026-07-08 00:16:04,003 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2239, Val Loss: 0.5898, Train Acc: 0.9772, Val Acc: 0.7718
2026-07-08 00:16:18,386 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2104, Val Loss: 0.6179, Train Acc: 0.9772, Val Acc: 0.5973
2026-07-08 00:16:32,780 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.1948, Val Loss: 0.6363, Train Acc: 0.9810, Val Acc: 0.5839
2026-07-08 00:16:47,152 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.1691, Val Loss: 0.4814, Train Acc: 0.9867, Val Acc: 0.8255
2026-07-08 00:17:01,530 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.1811, Val Loss: 0.6061, Train Acc: 0.9848, Val Acc: 0.6107
2026-07-08 00:17:15,903 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.1657, Val Loss: 0.5200, Train Acc: 0.9810, Val Acc: 0.7248
2026-07-08 00:17:16,036 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-08 00:17:30,436 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1632, Val Loss: 0.7477, Train Acc: 0.9791, Val Acc: 0.5772
2026-07-08 00:17:44,820 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1430, Val Loss: 0.6611, Train Acc: 0.9829, Val Acc: 0.5705
2026-07-08 00:17:59,200 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1345, Val Loss: 0.5500, Train Acc: 0.9848, Val Acc: 0.6376
2026-07-08 00:18:13,588 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1346, Val Loss: 0.5634, Train Acc: 0.9791, Val Acc: 0.6174
2026-07-08 00:18:27,966 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1259, Val Loss: 0.5657, Train Acc: 0.9810, Val Acc: 0.6577
2026-07-08 00:18:42,350 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1425, Val Loss: 0.9816, Train Acc: 0.9829, Val Acc: 0.5101
2026-07-08 00:18:56,733 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1162, Val Loss: 0.5702, Train Acc: 0.9829, Val Acc: 0.6376
2026-07-08 00:18:56,864 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-08 00:19:11,263 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1083, Val Loss: 0.4838, Train Acc: 0.9848, Val Acc: 0.8121
2026-07-08 00:19:25,655 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.1057, Val Loss: 0.4557, Train Acc: 0.9848, Val Acc: 0.7651
2026-07-08 00:19:40,058 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1055, Val Loss: 0.5828, Train Acc: 0.9829, Val Acc: 0.6242
2026-07-08 00:19:54,458 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.1187, Val Loss: 0.4799, Train Acc: 0.9848, Val Acc: 0.8255
2026-07-08 00:20:08,855 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.0932, Val Loss: 0.4705, Train Acc: 0.9848, Val Acc: 0.7718
2026-07-08 00:20:23,260 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.0860, Val Loss: 0.4911, Train Acc: 0.9810, Val Acc: 0.7315
2026-07-08 00:20:37,639 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.0840, Val Loss: 0.4949, Train Acc: 0.9848, Val Acc: 0.7919
2026-07-08 00:20:37,773 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-08 00:20:52,170 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0952, Val Loss: 0.4501, Train Acc: 0.9848, Val Acc: 0.8121
2026-07-08 00:21:06,673 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.0929, Val Loss: 0.8012, Train Acc: 0.9867, Val Acc: 0.5705
2026-07-08 00:21:07,362 - INFO - [Train Regular Models] Run 2/3 - Test Loss: 0.5541, Test Accuracy: 0.6598
2026-07-08 00:21:07,456 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_2.pt
2026-07-08 00:21:07,721 - INFO - [Train Regular Models] Training model 3/3...
2026-07-08 00:21:22,193 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5572, Val Loss: 0.8566, Train Acc: 0.6863, Val Acc: 0.4631
2026-07-08 00:21:36,577 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4454, Val Loss: 0.7267, Train Acc: 0.8080, Val Acc: 0.4832
2026-07-08 00:21:50,972 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.3922, Val Loss: 0.8255, Train Acc: 0.8783, Val Acc: 0.4631
2026-07-08 00:22:05,373 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3813, Val Loss: 0.9995, Train Acc: 0.8783, Val Acc: 0.4631
2026-07-08 00:22:19,769 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3363, Val Loss: 0.7541, Train Acc: 0.9544, Val Acc: 0.4765
2026-07-08 00:22:34,154 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.3080, Val Loss: 0.6512, Train Acc: 0.9601, Val Acc: 0.6040
2026-07-08 00:22:48,542 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2923, Val Loss: 0.7748, Train Acc: 0.9734, Val Acc: 0.4899
2026-07-08 00:22:48,682 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-08 00:23:03,070 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2716, Val Loss: 0.5976, Train Acc: 0.9734, Val Acc: 0.5906
2026-07-08 00:23:17,462 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2554, Val Loss: 0.6406, Train Acc: 0.9829, Val Acc: 0.5839
2026-07-08 00:23:31,848 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2467, Val Loss: 0.6147, Train Acc: 0.9753, Val Acc: 0.6644
2026-07-08 00:23:46,223 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.2402, Val Loss: 0.6437, Train Acc: 0.9829, Val Acc: 0.5503
2026-07-08 00:24:00,620 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.2135, Val Loss: 0.7625, Train Acc: 0.9791, Val Acc: 0.5235
2026-07-08 00:24:15,009 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.2046, Val Loss: 0.6169, Train Acc: 0.9848, Val Acc: 0.5705
2026-07-08 00:24:29,404 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.2064, Val Loss: 0.5363, Train Acc: 0.9658, Val Acc: 0.7315
2026-07-08 00:24:29,536 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-08 00:24:43,939 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1911, Val Loss: 0.5674, Train Acc: 0.9772, Val Acc: 0.6577
2026-07-08 00:24:58,322 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1871, Val Loss: 0.6471, Train Acc: 0.9753, Val Acc: 0.5906
2026-07-08 00:25:12,708 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1953, Val Loss: 0.7634, Train Acc: 0.9753, Val Acc: 0.4966
2026-07-08 00:25:27,097 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1603, Val Loss: 0.6293, Train Acc: 0.9867, Val Acc: 0.5772
2026-07-08 00:25:41,487 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1702, Val Loss: 0.7457, Train Acc: 0.9734, Val Acc: 0.5369
2026-07-08 00:25:55,859 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1605, Val Loss: 0.6624, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-08 00:26:10,248 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1508, Val Loss: 0.5531, Train Acc: 0.9772, Val Acc: 0.6376
2026-07-08 00:26:10,382 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-08 00:26:24,789 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1414, Val Loss: 0.5919, Train Acc: 0.9848, Val Acc: 0.6174
2026-07-08 00:26:39,199 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.1222, Val Loss: 0.5774, Train Acc: 0.9829, Val Acc: 0.6309
2026-07-08 00:26:53,587 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1249, Val Loss: 0.6473, Train Acc: 0.9829, Val Acc: 0.5705
2026-07-08 00:27:07,976 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.1343, Val Loss: 0.4817, Train Acc: 0.9734, Val Acc: 0.7718
2026-07-08 00:27:22,372 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.1164, Val Loss: 0.6104, Train Acc: 0.9829, Val Acc: 0.6107
2026-07-08 00:27:36,769 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.1146, Val Loss: 0.5914, Train Acc: 0.9848, Val Acc: 0.6040
2026-07-08 00:27:51,166 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.1126, Val Loss: 0.5165, Train Acc: 0.9829, Val Acc: 0.8188
2026-07-08 00:27:51,300 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-08 00:28:05,702 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0989, Val Loss: 0.5046, Train Acc: 0.9867, Val Acc: 0.8054
2026-07-08 00:28:20,085 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.0900, Val Loss: 0.5904, Train Acc: 0.9867, Val Acc: 0.6107
2026-07-08 00:28:20,765 - INFO - [Train Regular Models] Run 3/3 - Test Loss: 0.4318, Test Accuracy: 0.7216
2026-07-08 00:28:20,862 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_3.pt
2026-07-08 00:28:21,142 - INFO - [Train Regular Models] All models trained and saved successfully.
2026-07-08 00:28:21,143 - INFO - [Train Regular Models] Task completed.
2026-07-08 00:28:21,144 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-07-08 00:28:21,145 - INFO - [Train Bayesian Models] Starting bayesian model training...
2026-07-08 00:28:21,145 - INFO - [Train Bayesian Models] Intermediate models will be saved to outputs/exp1/bayesian_models/intermediate_models
2026-07-08 00:28:21,146 - INFO - [Train Bayesian Models] Training model 1/3...
2026-07-08 00:28:21,403 - ERROR - Pipeline failed: module 'bayesian_torch.layers' has no attribute 'Conv3dReparameterizationReparameterization'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 256, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_bayesian.py", line 91, in train_bayesian_task
    dnn_to_bnn(model, bnn_prior_parameters)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/bayesian_torch/models/dnn_to_bnn.py", line 130, in dnn_to_bnn
    dnn_to_bnn(m._modules[name], bnn_prior_parameters)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/bayesian_torch/models/dnn_to_bnn.py", line 130, in dnn_to_bnn
    dnn_to_bnn(m._modules[name], bnn_prior_parameters)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/bayesian_torch/models/dnn_to_bnn.py", line 130, in dnn_to_bnn
    dnn_to_bnn(m._modules[name], bnn_prior_parameters)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/bayesian_torch/models/dnn_to_bnn.py", line 135, in dnn_to_bnn
    bnn_conv_layer(
    ~~~~~~~~~~~~~~^
        bnn_prior_parameters,
        ^^^^^^^^^^^^^^^^^^^^^
        m._modules[name]))
        ^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/bayesian_torch/models/dnn_to_bnn.py", line 78, in bnn_conv_layer
    layer_fn = getattr(bayesian_layers, layer_type)  # Get BNN layer
AttributeError: module 'bayesian_torch.layers' has no attribute 'Conv3dReparameterizationReparameterization'
2026-07-08 00:28:21,404 - ERROR - [SYSTEM] ERROR: module 'bayesian_torch.layers' has no attribute 'Conv3dReparameterizationReparameterization'
2026-07-08 00:36:30,191 - INFO - [SYSTEM] Application initialized and ready.
2026-07-08 00:36:36,319 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:36:39,288 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-08 00:36:39,291 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-08 00:36:39,291 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-08 00:36:41,890 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-08 00:36:41,891 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-08 00:36:41,895 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-08 00:36:41,896 - INFO - [Train Regular Models] Starting normal model training...
2026-07-08 00:36:41,896 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-08 00:36:41,897 - INFO - [Train Regular Models] Training model 1/1...
2026-07-08 00:36:56,978 - INFO - [Train Regular Models] Epoch [1/30], Train Loss: 0.5711, Val Loss: 0.6639, Train Acc: 0.7072, Val Acc: 0.7248
2026-07-08 00:37:11,164 - INFO - [Train Regular Models] Epoch [2/30], Train Loss: 0.4657, Val Loss: 0.7017, Train Acc: 0.8137, Val Acc: 0.5302
2026-07-08 00:37:25,382 - INFO - [Train Regular Models] Epoch [3/30], Train Loss: 0.4026, Val Loss: 0.7230, Train Acc: 0.8954, Val Acc: 0.5034
2026-07-08 00:37:39,690 - INFO - [Train Regular Models] Epoch [4/30], Train Loss: 0.3698, Val Loss: 0.6492, Train Acc: 0.9221, Val Acc: 0.5570
2026-07-08 00:37:53,994 - INFO - [Train Regular Models] Epoch [5/30], Train Loss: 0.3217, Val Loss: 0.5782, Train Acc: 0.9620, Val Acc: 0.7651
2026-07-08 00:38:08,334 - INFO - [Train Regular Models] Epoch [6/30], Train Loss: 0.2877, Val Loss: 0.7763, Train Acc: 0.9734, Val Acc: 0.5034
2026-07-08 00:38:22,703 - INFO - [Train Regular Models] Epoch [7/30], Train Loss: 0.2673, Val Loss: 0.6497, Train Acc: 0.9810, Val Acc: 0.5503
2026-07-08 00:38:22,799 - INFO - [Train Regular Models] Model saved at epoch 7
2026-07-08 00:38:37,181 - INFO - [Train Regular Models] Epoch [8/30], Train Loss: 0.2526, Val Loss: 0.7038, Train Acc: 0.9810, Val Acc: 0.5436
2026-07-08 00:38:51,538 - INFO - [Train Regular Models] Epoch [9/30], Train Loss: 0.2446, Val Loss: 0.5974, Train Acc: 0.9791, Val Acc: 0.5973
2026-07-08 00:39:05,908 - INFO - [Train Regular Models] Epoch [10/30], Train Loss: 0.2381, Val Loss: 0.6177, Train Acc: 0.9715, Val Acc: 0.6309
2026-07-08 00:39:20,267 - INFO - [Train Regular Models] Epoch [11/30], Train Loss: 0.2046, Val Loss: 0.6750, Train Acc: 0.9810, Val Acc: 0.5638
2026-07-08 00:39:34,624 - INFO - [Train Regular Models] Epoch [12/30], Train Loss: 0.2025, Val Loss: 0.7757, Train Acc: 0.9772, Val Acc: 0.5302
2026-07-08 00:39:48,998 - INFO - [Train Regular Models] Epoch [13/30], Train Loss: 0.1668, Val Loss: 0.6402, Train Acc: 0.9867, Val Acc: 0.5839
2026-07-08 00:40:03,347 - INFO - [Train Regular Models] Epoch [14/30], Train Loss: 0.1760, Val Loss: 0.6179, Train Acc: 0.9772, Val Acc: 0.5906
2026-07-08 00:40:03,439 - INFO - [Train Regular Models] Model saved at epoch 14
2026-07-08 00:40:17,808 - INFO - [Train Regular Models] Epoch [15/30], Train Loss: 0.1504, Val Loss: 0.8002, Train Acc: 0.9848, Val Acc: 0.5503
2026-07-08 00:40:32,168 - INFO - [Train Regular Models] Epoch [16/30], Train Loss: 0.1392, Val Loss: 0.7660, Train Acc: 0.9867, Val Acc: 0.5302
2026-07-08 00:40:46,537 - INFO - [Train Regular Models] Epoch [17/30], Train Loss: 0.1379, Val Loss: 0.7277, Train Acc: 0.9886, Val Acc: 0.5705
2026-07-08 00:41:00,957 - INFO - [Train Regular Models] Epoch [18/30], Train Loss: 0.1302, Val Loss: 0.5244, Train Acc: 0.9810, Val Acc: 0.6846
2026-07-08 00:41:15,375 - INFO - [Train Regular Models] Epoch [19/30], Train Loss: 0.1323, Val Loss: 0.6825, Train Acc: 0.9829, Val Acc: 0.5705
2026-07-08 00:41:29,777 - INFO - [Train Regular Models] Epoch [20/30], Train Loss: 0.1227, Val Loss: 0.5189, Train Acc: 0.9791, Val Acc: 0.7785
2026-07-08 00:41:44,190 - INFO - [Train Regular Models] Epoch [21/30], Train Loss: 0.1184, Val Loss: 0.6989, Train Acc: 0.9867, Val Acc: 0.5705
2026-07-08 00:41:44,280 - INFO - [Train Regular Models] Model saved at epoch 21
2026-07-08 00:41:58,694 - INFO - [Train Regular Models] Epoch [22/30], Train Loss: 0.1342, Val Loss: 1.0438, Train Acc: 0.9715, Val Acc: 0.4966
2026-07-08 00:42:13,095 - INFO - [Train Regular Models] Epoch [23/30], Train Loss: 0.0966, Val Loss: 0.7316, Train Acc: 0.9829, Val Acc: 0.5503
2026-07-08 00:42:27,493 - INFO - [Train Regular Models] Epoch [24/30], Train Loss: 0.1013, Val Loss: 0.6301, Train Acc: 0.9867, Val Acc: 0.6107
2026-07-08 00:42:41,905 - INFO - [Train Regular Models] Epoch [25/30], Train Loss: 0.0966, Val Loss: 0.5920, Train Acc: 0.9829, Val Acc: 0.6376
2026-07-08 00:42:56,314 - INFO - [Train Regular Models] Epoch [26/30], Train Loss: 0.1000, Val Loss: 0.9743, Train Acc: 0.9810, Val Acc: 0.5235
2026-07-08 00:43:10,715 - INFO - [Train Regular Models] Epoch [27/30], Train Loss: 0.1022, Val Loss: 0.6421, Train Acc: 0.9810, Val Acc: 0.5906
2026-07-08 00:43:25,113 - INFO - [Train Regular Models] Epoch [28/30], Train Loss: 0.0837, Val Loss: 0.6173, Train Acc: 0.9867, Val Acc: 0.6174
2026-07-08 00:43:25,206 - INFO - [Train Regular Models] Model saved at epoch 28
2026-07-08 00:43:39,594 - INFO - [Train Regular Models] Epoch [29/30], Train Loss: 0.0918, Val Loss: 0.5776, Train Acc: 0.9791, Val Acc: 0.6510
2026-07-08 00:43:54,004 - INFO - [Train Regular Models] Epoch [30/30], Train Loss: 0.0830, Val Loss: 0.5306, Train Acc: 0.9829, Val Acc: 0.7181
2026-07-08 00:43:54,707 - INFO - [Train Regular Models] Run 1/1 - Test Loss: 0.4646, Test Accuracy: 0.8557
2026-07-08 00:43:54,808 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_1.pt
2026-07-08 00:43:55,070 - INFO - [Train Regular Models] All models trained and saved successfully.
2026-07-08 00:43:55,071 - INFO - [Train Regular Models] Task completed.
2026-07-08 00:43:55,072 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-07-08 00:43:55,073 - INFO - [Train Bayesian Models] Starting bayesian model training...
2026-07-08 00:43:55,073 - INFO - [Train Bayesian Models] Intermediate models will be saved to outputs/exp1/bayesian_models/intermediate_models
2026-07-08 00:43:55,074 - INFO - [Train Bayesian Models] Training model 1/1...
2026-07-08 00:43:55,324 - ERROR - Pipeline failed: module 'bayesian_torch.layers' has no attribute 'Conv3dReparameterizationReparameterization'
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 256, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_bayesian.py", line 91, in train_bayesian_task
    dnn_to_bnn_mod(model, bnn_prior_parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dnn_mod.py", line 132, in dnn_to_bnn_mod
    dnn_to_bnn_mod(m._modules[name], bnn_prior_parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dnn_mod.py", line 132, in dnn_to_bnn_mod
    dnn_to_bnn_mod(m._modules[name], bnn_prior_parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dnn_mod.py", line 132, in dnn_to_bnn_mod
    dnn_to_bnn_mod(m._modules[name], bnn_prior_parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dnn_mod.py", line 134, in dnn_to_bnn_mod
    setattr(m, name, bnn_conv_layer(bnn_prior_parameters, m._modules[name]))
                     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/dnn_mod.py", line 76, in bnn_conv_layer
    layer_fn = getattr(bayesian_layers, layer_type)  # Get BNN layer
AttributeError: module 'bayesian_torch.layers' has no attribute 'Conv3dReparameterizationReparameterization'
2026-07-08 00:43:55,325 - ERROR - [SYSTEM] ERROR: module 'bayesian_torch.layers' has no attribute 'Conv3dReparameterizationReparameterization'
2026-07-08 00:45:53,427 - INFO - [SYSTEM] Application initialized and ready.
2026-07-08 00:46:16,234 - INFO - [SYSTEM] Copied default config to ./ouputs/exp1/config.toml. Ready for editing!
2026-07-08 00:46:34,286 - INFO - [SYSTEM] Successfully loaded config from ./ouputs/exp1/config.toml
2026-07-08 00:46:35,839 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-08 00:46:35,842 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-08 00:46:35,842 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-08 00:46:38,425 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-08 00:46:38,430 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-08 00:46:38,432 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-08 00:46:38,432 - INFO - [Train Regular Models] Starting normal model training...
2026-07-08 00:46:38,432 - INFO - [Train Regular Models] Intermediate models will be saved to ouputs/exp1/normal_models/intermediate_models
2026-07-08 00:46:38,433 - INFO - [Train Regular Models] Training model 1/1...
2026-07-08 00:46:53,542 - INFO - [Train Regular Models] Epoch [1/1], Train Loss: 0.5775, Val Loss: 0.6650, Train Acc: 0.7357, Val Acc: 0.7047
2026-07-08 00:46:54,233 - INFO - [Train Regular Models] Run 1/1 - Test Loss: 0.6520, Test Accuracy: 0.7732
2026-07-08 00:46:54,334 - INFO - [Train Regular Models] Model saved to ouputs/exp1/normal_models/model_1.pt
2026-07-08 00:46:54,591 - INFO - [Train Regular Models] All models trained and saved successfully.
2026-07-08 00:46:54,592 - INFO - [Train Regular Models] Task completed.
2026-07-08 00:46:54,594 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-07-08 00:46:54,595 - INFO - [Train Bayesian Models] Starting bayesian model training...
2026-07-08 00:46:54,595 - INFO - [Train Bayesian Models] Intermediate models will be saved to ouputs/exp1/bayesian_models/intermediate_models
2026-07-08 00:46:54,596 - INFO - [Train Bayesian Models] Training model 1/1...
2026-07-08 00:46:55,035 - ERROR - Pipeline failed: Expected all tensors to be on the same device, but got weight is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__native_batch_norm)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 256, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_bayesian.py", line 98, in train_bayesian_task
    model, history = tn.train_model_bayesian(
                     ~~~~~~~~~~~~~~~~~~~~~~~^
        log=log,
        ^^^^^^^^
    ...<10 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 454, in train_model_bayesian
    train_loss, val_loss, train_acc, val_acc = train_epoch_bayesian(
                                               ~~~~~~~~~~~~~~~~~~~~^
        model,
        ^^^^^^
    ...<8 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 372, in train_epoch_bayesian
    outputs = model((mri, xls))
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 66, in forward
    image_out = self.image_section(image_data)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 34, in forward
    x = self.cnv1(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 119, in forward
    a = self.norm(a)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/batchnorm.py", line 210, in forward
    return F.batch_norm(
           ~~~~~~~~~~~~^
        input,
        ^^^^^^
    ...<11 lines>...
        self.eps,
        ^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/functional.py", line 2850, in batch_norm
    return torch.batch_norm(
           ~~~~~~~~~~~~~~~~^
        input,
        ^^^^^^
    ...<7 lines>...
        torch.backends.cudnn.enabled,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
RuntimeError: Expected all tensors to be on the same device, but got weight is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__native_batch_norm)
2026-07-08 00:46:55,037 - ERROR - [SYSTEM] ERROR: Expected all tensors to be on the same device, but got weight is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__native_batch_norm)
2026-07-08 00:50:01,319 - INFO - [SYSTEM] Application initialized and ready.
2026-07-08 00:50:09,578 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-08 00:50:24,023 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:50:25,887 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:50:36,955 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:50:53,802 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:50:55,008 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:51:00,546 - INFO - [SYSTEM] Application initialized and ready.
2026-07-08 00:51:08,097 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:51:27,522 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:51:28,532 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:51:35,020 - INFO - [SYSTEM] Successfully loaded config from ./outputs/exp1/config.toml
2026-07-08 00:51:36,630 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-08 00:51:36,642 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-08 00:51:36,643 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-08 00:51:39,204 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-08 00:51:39,208 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-08 00:51:39,210 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-08 00:51:39,211 - INFO - [Train Regular Models] Starting normal model training...
2026-07-08 00:51:39,211 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-08 00:51:39,212 - INFO - [Train Regular Models] Training model 1/1...
2026-07-08 00:51:54,318 - INFO - [Train Regular Models] Epoch [1/1], Train Loss: 0.5990, Val Loss: 0.6624, Train Acc: 0.6787, Val Acc: 0.5168
2026-07-08 00:51:55,011 - INFO - [Train Regular Models] Run 1/1 - Test Loss: 0.6625, Test Accuracy: 0.5979
2026-07-08 00:51:55,118 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_1.pt
2026-07-08 00:51:55,373 - INFO - [Train Regular Models] All models trained and saved successfully.
2026-07-08 00:51:55,374 - INFO - [Train Regular Models] Task completed.
2026-07-08 00:51:55,376 - INFO - [SYSTEM] Starting Task 3/5: Train Bayesian Models
2026-07-08 00:51:55,376 - INFO - [Train Bayesian Models] Starting bayesian model training...
2026-07-08 00:51:55,376 - INFO - [Train Bayesian Models] Intermediate models will be saved to outputs/exp1/bayesian_models/intermediate_models
2026-07-08 00:51:55,377 - INFO - [Train Bayesian Models] Training model 1/1...
2026-07-08 00:52:10,270 - INFO - [Train Bayesian Models] Epoch [1/1], Train Loss: 2.6497, Val Loss: 2.7953, Train Acc: 0.5589, Val Acc: 0.4631
2026-07-08 00:52:10,975 - INFO - [Train Bayesian Models] Run 1/1 - Test Loss: 3.2049, Test Accuracy: 0.6392
2026-07-08 00:52:11,118 - INFO - [Train Bayesian Models] Model saved to outputs/exp1/bayesian_models/model_1.pt
2026-07-08 00:52:11,352 - INFO - [Train Bayesian Models] All Bayesian models trained and saved successfully.
2026-07-08 00:52:11,353 - INFO - [Train Bayesian Models] Task completed.
2026-07-08 00:52:11,354 - INFO - [SYSTEM] Starting Task 4/5: Evaluate Regular Models
2026-07-08 00:52:11,355 - INFO - [Evaluate Regular Models] Initializing process...
2026-07-08 00:52:12,962 - INFO - [Evaluate Regular Models] Halfway through current task execution...
2026-07-08 00:52:14,041 - INFO - [Evaluate Regular Models] Task completed.
2026-07-08 00:52:14,043 - INFO - [SYSTEM] Starting Task 5/5: Evaluate Bayesian Models
2026-07-08 00:52:14,044 - INFO - [Evaluate Bayesian Models] Initializing process...
2026-07-08 00:52:15,667 - INFO - [Evaluate Bayesian Models] Halfway through current task execution...
2026-07-08 00:52:16,749 - INFO - [Evaluate Bayesian Models] Task completed.
2026-07-08 00:52:16,751 - INFO - [SYSTEM] All tasks finished successfully!
2026-07-08 00:52:49,209 - INFO - [SYSTEM] Application initialized and ready.
2026-07-08 00:52:54,655 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-08 00:52:57,639 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-08 00:52:57,642 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-08 00:52:57,643 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-08 00:53:00,242 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-08 00:53:00,243 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-08 00:53:00,248 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-08 00:53:00,249 - INFO - [Train Regular Models] Starting normal model training...
2026-07-08 00:53:00,249 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-08 00:53:00,250 - INFO - [Train Regular Models] Training model 1/30...
2026-07-08 00:53:15,384 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5373, Val Loss: 0.6979, Train Acc: 0.7757, Val Acc: 0.4966
2026-07-08 00:53:29,648 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4383, Val Loss: 0.7576, Train Acc: 0.8726, Val Acc: 0.4765
2026-07-08 00:53:43,937 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.4020, Val Loss: 0.6847, Train Acc: 0.9335, Val Acc: 0.5369
2026-07-08 00:53:58,288 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3858, Val Loss: 0.6350, Train Acc: 0.9411, Val Acc: 0.6174
2026-07-08 00:54:12,641 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3349, Val Loss: 0.6205, Train Acc: 0.9810, Val Acc: 0.6376
2026-07-08 00:54:27,023 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.3245, Val Loss: 0.7107, Train Acc: 0.9715, Val Acc: 0.5369
2026-07-08 00:54:27,118 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 00:54:41,514 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.3018, Val Loss: 0.5673, Train Acc: 0.9886, Val Acc: 0.6443
2026-07-08 00:54:55,906 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2890, Val Loss: 0.7526, Train Acc: 0.9677, Val Acc: 0.5369
2026-07-08 00:55:10,286 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2688, Val Loss: 0.6448, Train Acc: 0.9791, Val Acc: 0.5973
2026-07-08 00:55:24,653 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2461, Val Loss: 0.8519, Train Acc: 0.9848, Val Acc: 0.5101
2026-07-08 00:55:39,031 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2664, Val Loss: 0.5699, Train Acc: 0.9620, Val Acc: 0.6980
2026-07-08 00:55:53,414 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2257, Val Loss: 0.6403, Train Acc: 0.9791, Val Acc: 0.5839
2026-07-08 00:55:53,504 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 00:56:07,874 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.2107, Val Loss: 0.6762, Train Acc: 0.9848, Val Acc: 0.5570
2026-07-08 00:56:22,254 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1982, Val Loss: 0.7457, Train Acc: 0.9791, Val Acc: 0.5302
2026-07-08 00:56:36,620 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.2013, Val Loss: 0.7011, Train Acc: 0.9829, Val Acc: 0.5570
2026-07-08 00:56:51,003 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1811, Val Loss: 0.5839, Train Acc: 0.9810, Val Acc: 0.6443
2026-07-08 00:57:05,373 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1703, Val Loss: 0.5917, Train Acc: 0.9810, Val Acc: 0.6242
2026-07-08 00:57:19,748 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1627, Val Loss: 0.5703, Train Acc: 0.9772, Val Acc: 0.6644
2026-07-08 00:57:19,840 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 00:57:34,225 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1401, Val Loss: 0.6857, Train Acc: 0.9905, Val Acc: 0.5906
2026-07-08 00:57:48,704 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1415, Val Loss: 0.5536, Train Acc: 0.9791, Val Acc: 0.6376
2026-07-08 00:58:03,088 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1345, Val Loss: 0.4856, Train Acc: 0.9867, Val Acc: 0.7517
2026-07-08 00:58:17,472 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1262, Val Loss: 0.4994, Train Acc: 0.9829, Val Acc: 0.7852
2026-07-08 00:58:31,844 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1120, Val Loss: 0.6324, Train Acc: 0.9867, Val Acc: 0.6107
2026-07-08 00:58:46,219 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1253, Val Loss: 0.6264, Train Acc: 0.9829, Val Acc: 0.6107
2026-07-08 00:58:46,314 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 00:59:00,697 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1001, Val Loss: 0.6807, Train Acc: 0.9848, Val Acc: 0.5772
2026-07-08 00:59:01,388 - INFO - [Train Regular Models] Run 1/30 - Test Loss: 0.5012, Test Accuracy: 0.6907
2026-07-08 00:59:01,488 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_1.pt
2026-07-08 00:59:01,757 - INFO - [Train Regular Models] Training model 2/30...
2026-07-08 00:59:16,203 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.6134, Val Loss: 0.6832, Train Acc: 0.7586, Val Acc: 0.5638
2026-07-08 00:59:30,554 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.5120, Val Loss: 0.6103, Train Acc: 0.9030, Val Acc: 0.7383
2026-07-08 00:59:44,925 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.4746, Val Loss: 0.6348, Train Acc: 0.9373, Val Acc: 0.6510
2026-07-08 00:59:59,330 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.4354, Val Loss: 0.5920, Train Acc: 0.9677, Val Acc: 0.7919
2026-07-08 01:00:11,477 - ERROR - Pipeline failed: App is not running
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 256, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 90, in train_normal_task
    model, history = tn.train_model(
                     ~~~~~~~~~~~~~~^
        log=log,
        ^^^^^^^^
    ...<9 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 248, in train_model
    train_loss, val_loss, train_acc, val_acc = train_epoch(
                                               ~~~~~~~~~~~^
        model,
        ^^^^^^
    ...<7 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 174, in train_epoch
    train_progress.update(total=len(train_loader), advance=1)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/util/progress.py", line 64, in update
    self.app.call_from_thread(_do_update)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/textual/app.py", line 1819, in call_from_thread
    raise RuntimeError("App is not running")
RuntimeError: App is not running
2026-07-08 01:00:11,478 - ERROR - [SYSTEM] ERROR: App is not running
2026-07-08 01:00:57,572 - INFO - [SYSTEM] Application initialized and ready.
2026-07-08 01:01:03,263 - INFO - [SYSTEM] Copied default config to ./outputs/exp1/config.toml. Ready for editing!
2026-07-08 01:01:07,863 - INFO - [SYSTEM] Initiating Scenario: 1. Train, Evaluate, & Noise Analysis
2026-07-08 01:01:07,866 - INFO - [SYSTEM] Starting Task 1/5: Load Image and ADNIMERGE
2026-07-08 01:01:07,866 - INFO - [Load Image and ADNIMERGE] Loading Files
2026-07-08 01:01:10,532 - INFO - [Load Image and ADNIMERGE] Dataloaders initalized
2026-07-08 01:01:10,537 - INFO - [Load Image and ADNIMERGE] Task completed.
2026-07-08 01:01:10,538 - INFO - [SYSTEM] Starting Task 2/5: Train Regular Models
2026-07-08 01:01:10,539 - INFO - [Train Regular Models] Starting normal model training...
2026-07-08 01:01:10,539 - INFO - [Train Regular Models] Intermediate models will be saved to outputs/exp1/normal_models/intermediate_models
2026-07-08 01:01:10,540 - INFO - [Train Regular Models] Training model 1/30...
2026-07-08 01:01:25,669 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5300, Val Loss: 0.7628, Train Acc: 0.7985, Val Acc: 0.4698
2026-07-08 01:01:39,932 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.3715, Val Loss: 0.7620, Train Acc: 0.9087, Val Acc: 0.4966
2026-07-08 01:01:54,234 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3557, Val Loss: 0.6723, Train Acc: 0.9068, Val Acc: 0.5570
2026-07-08 01:02:08,548 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.2999, Val Loss: 0.9527, Train Acc: 0.9430, Val Acc: 0.4631
2026-07-08 01:02:22,938 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.2659, Val Loss: 0.7849, Train Acc: 0.9696, Val Acc: 0.4832
2026-07-08 01:02:37,339 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2204, Val Loss: 0.5281, Train Acc: 0.9810, Val Acc: 0.7852
2026-07-08 01:02:37,436 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:02:51,818 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2478, Val Loss: 0.9242, Train Acc: 0.9620, Val Acc: 0.4631
2026-07-08 01:03:06,222 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2405, Val Loss: 0.8734, Train Acc: 0.9772, Val Acc: 0.4698
2026-07-08 01:03:20,720 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.1978, Val Loss: 0.8342, Train Acc: 0.9829, Val Acc: 0.4966
2026-07-08 01:03:35,123 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.1710, Val Loss: 0.6356, Train Acc: 0.9810, Val Acc: 0.5906
2026-07-08 01:03:49,528 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.1511, Val Loss: 0.5789, Train Acc: 0.9867, Val Acc: 0.6779
2026-07-08 01:04:03,919 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1370, Val Loss: 0.8028, Train Acc: 0.9848, Val Acc: 0.5034
2026-07-08 01:04:04,010 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:04:18,422 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1273, Val Loss: 0.5179, Train Acc: 0.9791, Val Acc: 0.6913
2026-07-08 01:04:32,816 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1207, Val Loss: 0.4625, Train Acc: 0.9848, Val Acc: 0.7852
2026-07-08 01:04:47,212 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1181, Val Loss: 0.5121, Train Acc: 0.9848, Val Acc: 0.7718
2026-07-08 01:05:01,602 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1123, Val Loss: 0.4885, Train Acc: 0.9810, Val Acc: 0.7785
2026-07-08 01:05:16,002 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1106, Val Loss: 0.7487, Train Acc: 0.9848, Val Acc: 0.5638
2026-07-08 01:05:30,407 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.0957, Val Loss: 0.4101, Train Acc: 0.9829, Val Acc: 0.8456
2026-07-08 01:05:30,502 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:05:44,887 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.0936, Val Loss: 0.7903, Train Acc: 0.9867, Val Acc: 0.5436
2026-07-08 01:05:59,282 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.0880, Val Loss: 0.6095, Train Acc: 0.9905, Val Acc: 0.6376
2026-07-08 01:06:13,662 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.0928, Val Loss: 0.6065, Train Acc: 0.9829, Val Acc: 0.6174
2026-07-08 01:06:28,041 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.0829, Val Loss: 0.4587, Train Acc: 0.9848, Val Acc: 0.8054
2026-07-08 01:06:42,435 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.0774, Val Loss: 0.5372, Train Acc: 0.9848, Val Acc: 0.6577
2026-07-08 01:06:56,817 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.0699, Val Loss: 0.8077, Train Acc: 0.9848, Val Acc: 0.5638
2026-07-08 01:06:56,906 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:07:11,304 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.0909, Val Loss: 0.9999, Train Acc: 0.9810, Val Acc: 0.4966
2026-07-08 01:07:11,995 - INFO - [Train Regular Models] Run 1/30 - Test Loss: 0.6957, Test Accuracy: 0.6495
2026-07-08 01:07:12,094 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_1.pt
2026-07-08 01:07:12,356 - INFO - [Train Regular Models] Training model 2/30...
2026-07-08 01:07:26,841 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5411, Val Loss: 0.8264, Train Acc: 0.7586, Val Acc: 0.4631
2026-07-08 01:07:41,218 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4025, Val Loss: 0.7183, Train Acc: 0.8973, Val Acc: 0.4899
2026-07-08 01:07:55,589 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3528, Val Loss: 0.5507, Train Acc: 0.9297, Val Acc: 0.7584
2026-07-08 01:08:09,985 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3389, Val Loss: 0.6291, Train Acc: 0.9563, Val Acc: 0.5772
2026-07-08 01:08:24,375 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.2878, Val Loss: 0.6898, Train Acc: 0.9677, Val Acc: 0.5369
2026-07-08 01:08:38,755 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2615, Val Loss: 0.5385, Train Acc: 0.9734, Val Acc: 0.7919
2026-07-08 01:08:38,894 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:08:53,274 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2349, Val Loss: 0.7455, Train Acc: 0.9829, Val Acc: 0.5101
2026-07-08 01:09:07,656 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2136, Val Loss: 0.5736, Train Acc: 0.9772, Val Acc: 0.6376
2026-07-08 01:09:22,040 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2176, Val Loss: 0.5717, Train Acc: 0.9791, Val Acc: 0.6376
2026-07-08 01:09:36,418 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2045, Val Loss: 0.5419, Train Acc: 0.9810, Val Acc: 0.6846
2026-07-08 01:09:50,817 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2045, Val Loss: 0.7658, Train Acc: 0.9829, Val Acc: 0.5235
2026-07-08 01:10:05,200 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1783, Val Loss: 0.6879, Train Acc: 0.9791, Val Acc: 0.5638
2026-07-08 01:10:05,334 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:10:19,736 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1695, Val Loss: 0.5749, Train Acc: 0.9829, Val Acc: 0.7987
2026-07-08 01:10:34,114 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1835, Val Loss: 0.5658, Train Acc: 0.9829, Val Acc: 0.7450
2026-07-08 01:10:48,500 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1493, Val Loss: 0.5649, Train Acc: 0.9829, Val Acc: 0.7651
2026-07-08 01:11:02,874 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1629, Val Loss: 0.5003, Train Acc: 0.9791, Val Acc: 0.7785
2026-07-08 01:11:17,268 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1338, Val Loss: 0.5780, Train Acc: 0.9867, Val Acc: 0.6443
2026-07-08 01:11:31,662 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1239, Val Loss: 0.5313, Train Acc: 0.9829, Val Acc: 0.7450
2026-07-08 01:11:31,800 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:11:46,196 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1213, Val Loss: 0.5926, Train Acc: 0.9772, Val Acc: 0.6309
2026-07-08 01:12:00,581 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1209, Val Loss: 0.4510, Train Acc: 0.9791, Val Acc: 0.8456
2026-07-08 01:12:14,975 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1119, Val Loss: 0.5871, Train Acc: 0.9829, Val Acc: 0.6376
2026-07-08 01:12:29,348 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1067, Val Loss: 0.6883, Train Acc: 0.9848, Val Acc: 0.5839
2026-07-08 01:12:43,734 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1047, Val Loss: 0.6176, Train Acc: 0.9829, Val Acc: 0.6107
2026-07-08 01:12:58,123 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.0937, Val Loss: 0.6716, Train Acc: 0.9848, Val Acc: 0.5839
2026-07-08 01:12:58,259 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:13:12,657 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.0930, Val Loss: 0.4147, Train Acc: 0.9810, Val Acc: 0.8658
2026-07-08 01:13:13,349 - INFO - [Train Regular Models] Run 2/30 - Test Loss: 0.3562, Test Accuracy: 0.9072
2026-07-08 01:13:13,444 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_2.pt
2026-07-08 01:13:13,713 - INFO - [Train Regular Models] Training model 3/30...
2026-07-08 01:13:28,195 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5169, Val Loss: 0.6486, Train Acc: 0.7700, Val Acc: 0.5302
2026-07-08 01:13:42,584 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.3620, Val Loss: 0.5683, Train Acc: 0.9030, Val Acc: 0.7450
2026-07-08 01:13:56,997 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.2968, Val Loss: 0.5747, Train Acc: 0.9639, Val Acc: 0.6443
2026-07-08 01:14:11,377 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.2512, Val Loss: 0.5743, Train Acc: 0.9734, Val Acc: 0.6443
2026-07-08 01:14:25,767 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.2188, Val Loss: 0.6185, Train Acc: 0.9810, Val Acc: 0.5503
2026-07-08 01:14:40,168 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2150, Val Loss: 0.4868, Train Acc: 0.9772, Val Acc: 0.7315
2026-07-08 01:14:40,307 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:14:54,686 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2025, Val Loss: 0.6411, Train Acc: 0.9848, Val Acc: 0.5705
2026-07-08 01:15:09,097 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.1710, Val Loss: 0.6262, Train Acc: 0.9772, Val Acc: 0.5705
2026-07-08 01:15:23,494 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.1698, Val Loss: 0.5682, Train Acc: 0.9772, Val Acc: 0.6376
2026-07-08 01:15:37,887 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.1643, Val Loss: 0.6054, Train Acc: 0.9829, Val Acc: 0.6174
2026-07-08 01:15:52,286 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.1601, Val Loss: 0.7624, Train Acc: 0.9772, Val Acc: 0.5436
2026-07-08 01:16:06,682 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1421, Val Loss: 0.5078, Train Acc: 0.9753, Val Acc: 0.8188
2026-07-08 01:16:06,816 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:16:21,218 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1406, Val Loss: 0.5469, Train Acc: 0.9810, Val Acc: 0.6711
2026-07-08 01:16:35,601 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1237, Val Loss: 0.6250, Train Acc: 0.9829, Val Acc: 0.5906
2026-07-08 01:16:50,005 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1067, Val Loss: 0.5108, Train Acc: 0.9867, Val Acc: 0.7047
2026-07-08 01:17:04,483 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.0998, Val Loss: 0.4392, Train Acc: 0.9772, Val Acc: 0.8523
2026-07-08 01:17:18,887 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.0994, Val Loss: 0.4548, Train Acc: 0.9867, Val Acc: 0.7383
2026-07-08 01:17:33,284 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1006, Val Loss: 1.0407, Train Acc: 0.9848, Val Acc: 0.4832
2026-07-08 01:17:33,417 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:17:47,818 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.0943, Val Loss: 0.5732, Train Acc: 0.9772, Val Acc: 0.6242
2026-07-08 01:18:02,223 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.0869, Val Loss: 0.4378, Train Acc: 0.9848, Val Acc: 0.7919
2026-07-08 01:18:16,632 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.0871, Val Loss: 0.4648, Train Acc: 0.9848, Val Acc: 0.7987
2026-07-08 01:18:31,017 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.0814, Val Loss: 0.5271, Train Acc: 0.9905, Val Acc: 0.6779
2026-07-08 01:18:45,402 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.0826, Val Loss: 0.4377, Train Acc: 0.9810, Val Acc: 0.8121
2026-07-08 01:18:59,796 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.0809, Val Loss: 0.5062, Train Acc: 0.9810, Val Acc: 0.6846
2026-07-08 01:18:59,928 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:19:14,324 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.0781, Val Loss: 0.5860, Train Acc: 0.9829, Val Acc: 0.6242
2026-07-08 01:19:15,008 - INFO - [Train Regular Models] Run 3/30 - Test Loss: 0.4702, Test Accuracy: 0.7835
2026-07-08 01:19:15,098 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_3.pt
2026-07-08 01:19:15,363 - INFO - [Train Regular Models] Training model 4/30...
2026-07-08 01:19:29,838 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5427, Val Loss: 0.6841, Train Acc: 0.7700, Val Acc: 0.4698
2026-07-08 01:19:44,238 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4298, Val Loss: 0.6190, Train Acc: 0.8859, Val Acc: 0.7315
2026-07-08 01:19:58,630 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3742, Val Loss: 0.6703, Train Acc: 0.9525, Val Acc: 0.5101
2026-07-08 01:20:13,006 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3373, Val Loss: 0.6394, Train Acc: 0.9639, Val Acc: 0.8054
2026-07-08 01:20:27,385 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3187, Val Loss: 0.6481, Train Acc: 0.9639, Val Acc: 0.5570
2026-07-08 01:20:41,764 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.3023, Val Loss: 0.5691, Train Acc: 0.9753, Val Acc: 0.6242
2026-07-08 01:20:41,901 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:20:56,304 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2736, Val Loss: 0.7953, Train Acc: 0.9772, Val Acc: 0.4832
2026-07-08 01:21:10,706 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2645, Val Loss: 0.5759, Train Acc: 0.9772, Val Acc: 0.6309
2026-07-08 01:21:25,105 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2312, Val Loss: 0.5686, Train Acc: 0.9791, Val Acc: 0.6644
2026-07-08 01:21:39,499 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2250, Val Loss: 0.6956, Train Acc: 0.9772, Val Acc: 0.5503
2026-07-08 01:21:53,895 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2036, Val Loss: 0.6429, Train Acc: 0.9810, Val Acc: 0.5705
2026-07-08 01:22:08,291 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1982, Val Loss: 0.5864, Train Acc: 0.9810, Val Acc: 0.6510
2026-07-08 01:22:08,421 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:22:22,815 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1822, Val Loss: 0.6793, Train Acc: 0.9810, Val Acc: 0.5839
2026-07-08 01:22:37,218 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1730, Val Loss: 0.6120, Train Acc: 0.9829, Val Acc: 0.5772
2026-07-08 01:22:51,603 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1654, Val Loss: 0.6889, Train Acc: 0.9791, Val Acc: 0.5772
2026-07-08 01:23:06,103 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1508, Val Loss: 0.5751, Train Acc: 0.9810, Val Acc: 0.6309
2026-07-08 01:23:20,489 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1566, Val Loss: 0.6450, Train Acc: 0.9772, Val Acc: 0.5839
2026-07-08 01:23:34,886 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1415, Val Loss: 0.5272, Train Acc: 0.9867, Val Acc: 0.6913
2026-07-08 01:23:35,024 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:23:49,421 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1453, Val Loss: 0.7403, Train Acc: 0.9791, Val Acc: 0.5772
2026-07-08 01:24:03,813 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1355, Val Loss: 0.6999, Train Acc: 0.9810, Val Acc: 0.5638
2026-07-08 01:24:18,195 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1216, Val Loss: 0.5529, Train Acc: 0.9886, Val Acc: 0.6779
2026-07-08 01:24:32,571 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1248, Val Loss: 0.7021, Train Acc: 0.9867, Val Acc: 0.5705
2026-07-08 01:24:46,971 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1119, Val Loss: 0.7781, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-08 01:25:01,383 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1131, Val Loss: 0.6185, Train Acc: 0.9867, Val Acc: 0.6107
2026-07-08 01:25:01,516 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:25:15,917 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1055, Val Loss: 0.7148, Train Acc: 0.9829, Val Acc: 0.5705
2026-07-08 01:25:16,596 - INFO - [Train Regular Models] Run 4/30 - Test Loss: 0.5307, Test Accuracy: 0.6701
2026-07-08 01:25:16,694 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_4.pt
2026-07-08 01:25:16,962 - INFO - [Train Regular Models] Training model 5/30...
2026-07-08 01:25:31,446 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5741, Val Loss: 0.7368, Train Acc: 0.7319, Val Acc: 0.5101
2026-07-08 01:25:45,839 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4103, Val Loss: 0.5369, Train Acc: 0.8992, Val Acc: 0.7852
2026-07-08 01:26:00,235 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3421, Val Loss: 0.6047, Train Acc: 0.9373, Val Acc: 0.6510
2026-07-08 01:26:14,633 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.2881, Val Loss: 0.6568, Train Acc: 0.9696, Val Acc: 0.5638
2026-07-08 01:26:29,007 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.2555, Val Loss: 0.6403, Train Acc: 0.9829, Val Acc: 0.5369
2026-07-08 01:26:43,403 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2359, Val Loss: 0.5632, Train Acc: 0.9772, Val Acc: 0.6309
2026-07-08 01:26:43,539 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:26:57,934 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2152, Val Loss: 0.5543, Train Acc: 0.9810, Val Acc: 0.8121
2026-07-08 01:27:12,321 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2136, Val Loss: 0.5741, Train Acc: 0.9734, Val Acc: 0.7852
2026-07-08 01:27:26,717 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.1930, Val Loss: 0.4875, Train Acc: 0.9791, Val Acc: 0.8591
2026-07-08 01:27:41,104 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.1683, Val Loss: 0.5710, Train Acc: 0.9829, Val Acc: 0.6577
2026-07-08 01:27:55,512 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.1601, Val Loss: 0.5072, Train Acc: 0.9753, Val Acc: 0.7315
2026-07-08 01:28:09,901 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1580, Val Loss: 0.4940, Train Acc: 0.9791, Val Acc: 0.8591
2026-07-08 01:28:10,036 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:28:24,433 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1389, Val Loss: 0.5182, Train Acc: 0.9753, Val Acc: 0.8322
2026-07-08 01:28:38,818 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1320, Val Loss: 0.5652, Train Acc: 0.9829, Val Acc: 0.6107
2026-07-08 01:28:53,217 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1180, Val Loss: 0.4900, Train Acc: 0.9810, Val Acc: 0.7517
2026-07-08 01:29:07,694 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1120, Val Loss: 0.4352, Train Acc: 0.9848, Val Acc: 0.8322
2026-07-08 01:29:22,103 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1140, Val Loss: 0.4390, Train Acc: 0.9848, Val Acc: 0.8658
2026-07-08 01:29:36,508 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1119, Val Loss: 0.6699, Train Acc: 0.9791, Val Acc: 0.5705
2026-07-08 01:29:36,642 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:29:51,034 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.0996, Val Loss: 0.5842, Train Acc: 0.9829, Val Acc: 0.6242
2026-07-08 01:30:05,424 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1080, Val Loss: 0.6652, Train Acc: 0.9810, Val Acc: 0.5839
2026-07-08 01:30:19,820 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.0978, Val Loss: 0.5619, Train Acc: 0.9791, Val Acc: 0.6443
2026-07-08 01:30:34,218 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1016, Val Loss: 0.6455, Train Acc: 0.9829, Val Acc: 0.6040
2026-07-08 01:30:48,614 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.0991, Val Loss: 0.5913, Train Acc: 0.9753, Val Acc: 0.6174
2026-07-08 01:31:02,998 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.0901, Val Loss: 0.4383, Train Acc: 0.9886, Val Acc: 0.8456
2026-07-08 01:31:03,132 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:31:17,518 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.0853, Val Loss: 0.4694, Train Acc: 0.9848, Val Acc: 0.7383
2026-07-08 01:31:18,203 - INFO - [Train Regular Models] Run 5/30 - Test Loss: 0.3819, Test Accuracy: 0.8247
2026-07-08 01:31:18,292 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_5.pt
2026-07-08 01:31:18,563 - INFO - [Train Regular Models] Training model 6/30...
2026-07-08 01:31:33,051 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5164, Val Loss: 0.5925, Train Acc: 0.7966, Val Acc: 0.7852
2026-07-08 01:31:47,439 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4152, Val Loss: 0.6535, Train Acc: 0.9259, Val Acc: 0.5101
2026-07-08 01:32:01,840 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3611, Val Loss: 0.5305, Train Acc: 0.9563, Val Acc: 0.8591
2026-07-08 01:32:16,229 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3079, Val Loss: 0.5751, Train Acc: 0.9658, Val Acc: 0.8188
2026-07-08 01:32:30,612 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3010, Val Loss: 0.5695, Train Acc: 0.9715, Val Acc: 0.6980
2026-07-08 01:32:45,003 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2808, Val Loss: 0.5260, Train Acc: 0.9772, Val Acc: 0.8389
2026-07-08 01:32:45,141 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:32:59,538 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2579, Val Loss: 0.5604, Train Acc: 0.9791, Val Acc: 0.7248
2026-07-08 01:33:13,936 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2599, Val Loss: 0.7377, Train Acc: 0.9715, Val Acc: 0.5369
2026-07-08 01:33:28,354 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2371, Val Loss: 0.6875, Train Acc: 0.9791, Val Acc: 0.5503
2026-07-08 01:33:42,753 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2347, Val Loss: 0.6390, Train Acc: 0.9753, Val Acc: 0.5906
2026-07-08 01:33:57,158 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2066, Val Loss: 0.5616, Train Acc: 0.9753, Val Acc: 0.7047
2026-07-08 01:34:11,563 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1795, Val Loss: 0.4693, Train Acc: 0.9848, Val Acc: 0.8658
2026-07-08 01:34:11,703 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:34:26,104 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1965, Val Loss: 0.5545, Train Acc: 0.9715, Val Acc: 0.6644
2026-07-08 01:34:40,500 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1665, Val Loss: 0.5037, Train Acc: 0.9829, Val Acc: 0.8456
2026-07-08 01:34:54,895 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1588, Val Loss: 0.5272, Train Acc: 0.9810, Val Acc: 0.6577
2026-07-08 01:35:09,307 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1600, Val Loss: 0.6634, Train Acc: 0.9772, Val Acc: 0.5503
2026-07-08 01:35:23,705 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1397, Val Loss: 0.5380, Train Acc: 0.9734, Val Acc: 0.8054
2026-07-08 01:35:38,108 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1440, Val Loss: 0.5442, Train Acc: 0.9810, Val Acc: 0.6779
2026-07-08 01:35:38,243 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:35:52,636 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1216, Val Loss: 0.5735, Train Acc: 0.9810, Val Acc: 0.6107
2026-07-08 01:36:07,038 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1141, Val Loss: 0.5466, Train Acc: 0.9886, Val Acc: 0.7852
2026-07-08 01:36:21,428 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1282, Val Loss: 0.5997, Train Acc: 0.9772, Val Acc: 0.7383
2026-07-08 01:36:35,817 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1120, Val Loss: 0.5995, Train Acc: 0.9829, Val Acc: 0.6376
2026-07-08 01:36:50,201 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1136, Val Loss: 0.6835, Train Acc: 0.9772, Val Acc: 0.5503
2026-07-08 01:37:04,611 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1072, Val Loss: 0.8303, Train Acc: 0.9829, Val Acc: 0.5302
2026-07-08 01:37:04,745 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:37:19,140 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1054, Val Loss: 0.5021, Train Acc: 0.9791, Val Acc: 0.7181
2026-07-08 01:37:19,827 - INFO - [Train Regular Models] Run 6/30 - Test Loss: 0.4059, Test Accuracy: 0.8041
2026-07-08 01:37:19,920 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_6.pt
2026-07-08 01:37:20,193 - INFO - [Train Regular Models] Training model 7/30...
2026-07-08 01:37:34,700 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.4952, Val Loss: 0.9333, Train Acc: 0.7719, Val Acc: 0.4631
2026-07-08 01:37:49,087 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.3434, Val Loss: 0.9318, Train Acc: 0.9030, Val Acc: 0.4631
2026-07-08 01:38:03,489 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.2749, Val Loss: 0.6245, Train Acc: 0.9487, Val Acc: 0.6242
2026-07-08 01:38:17,891 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.2519, Val Loss: 0.7076, Train Acc: 0.9734, Val Acc: 0.5436
2026-07-08 01:38:32,292 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.2420, Val Loss: 0.6122, Train Acc: 0.9677, Val Acc: 0.6040
2026-07-08 01:38:46,678 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2132, Val Loss: 0.7147, Train Acc: 0.9734, Val Acc: 0.5503
2026-07-08 01:38:46,816 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:39:01,229 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.1928, Val Loss: 0.8075, Train Acc: 0.9829, Val Acc: 0.5101
2026-07-08 01:39:15,628 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.1737, Val Loss: 0.5032, Train Acc: 0.9791, Val Acc: 0.8389
2026-07-08 01:39:30,029 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.1793, Val Loss: 0.6098, Train Acc: 0.9810, Val Acc: 0.6174
2026-07-08 01:39:44,419 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.1620, Val Loss: 0.8713, Train Acc: 0.9753, Val Acc: 0.5168
2026-07-08 01:39:58,818 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.1442, Val Loss: 0.6082, Train Acc: 0.9810, Val Acc: 0.5839
2026-07-08 01:40:13,214 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1405, Val Loss: 0.5352, Train Acc: 0.9753, Val Acc: 0.7987
2026-07-08 01:40:13,350 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:40:27,755 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1344, Val Loss: 0.6029, Train Acc: 0.9829, Val Acc: 0.6040
2026-07-08 01:40:42,161 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1205, Val Loss: 0.6131, Train Acc: 0.9791, Val Acc: 0.5973
2026-07-08 01:40:56,555 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1122, Val Loss: 0.6026, Train Acc: 0.9848, Val Acc: 0.6107
2026-07-08 01:41:10,936 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1093, Val Loss: 0.5155, Train Acc: 0.9791, Val Acc: 0.7785
2026-07-08 01:41:25,337 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1208, Val Loss: 0.6976, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-08 01:41:39,737 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1344, Val Loss: 1.0305, Train Acc: 0.9791, Val Acc: 0.5034
2026-07-08 01:41:39,874 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:41:54,279 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1105, Val Loss: 0.8610, Train Acc: 0.9886, Val Acc: 0.5302
2026-07-08 01:42:08,664 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1144, Val Loss: 0.9194, Train Acc: 0.9791, Val Acc: 0.5369
2026-07-08 01:42:23,062 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1017, Val Loss: 0.5619, Train Acc: 0.9867, Val Acc: 0.6510
2026-07-08 01:42:37,445 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.0929, Val Loss: 0.5483, Train Acc: 0.9886, Val Acc: 0.7181
2026-07-08 01:42:51,845 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.0940, Val Loss: 0.5473, Train Acc: 0.9810, Val Acc: 0.6577
2026-07-08 01:43:06,234 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.0794, Val Loss: 0.5701, Train Acc: 0.9867, Val Acc: 0.6443
2026-07-08 01:43:06,369 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:43:20,770 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.0881, Val Loss: 0.5758, Train Acc: 0.9791, Val Acc: 0.6107
2026-07-08 01:43:21,465 - INFO - [Train Regular Models] Run 7/30 - Test Loss: 0.4716, Test Accuracy: 0.7216
2026-07-08 01:43:21,560 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_7.pt
2026-07-08 01:43:21,830 - INFO - [Train Regular Models] Training model 8/30...
2026-07-08 01:43:36,303 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5739, Val Loss: 0.6754, Train Acc: 0.7357, Val Acc: 0.5705
2026-07-08 01:43:50,673 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4614, Val Loss: 0.6468, Train Acc: 0.9087, Val Acc: 0.5638
2026-07-08 01:44:05,052 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.4142, Val Loss: 0.5501, Train Acc: 0.9335, Val Acc: 0.7919
2026-07-08 01:44:19,429 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3823, Val Loss: 0.6099, Train Acc: 0.9525, Val Acc: 0.5973
2026-07-08 01:44:33,802 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3473, Val Loss: 0.5656, Train Acc: 0.9639, Val Acc: 0.8188
2026-07-08 01:44:48,192 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.3123, Val Loss: 0.5210, Train Acc: 0.9772, Val Acc: 0.8054
2026-07-08 01:44:48,305 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:45:02,678 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2948, Val Loss: 0.5197, Train Acc: 0.9772, Val Acc: 0.7987
2026-07-08 01:45:17,038 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2799, Val Loss: 0.5473, Train Acc: 0.9677, Val Acc: 0.7047
2026-07-08 01:45:31,404 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2505, Val Loss: 0.5295, Train Acc: 0.9753, Val Acc: 0.8188
2026-07-08 01:45:45,786 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2431, Val Loss: 0.5406, Train Acc: 0.9772, Val Acc: 0.7919
2026-07-08 01:46:00,167 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2212, Val Loss: 0.5404, Train Acc: 0.9810, Val Acc: 0.8255
2026-07-08 01:46:14,538 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2129, Val Loss: 0.4794, Train Acc: 0.9753, Val Acc: 0.8591
2026-07-08 01:46:14,658 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:46:29,033 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1922, Val Loss: 0.5159, Train Acc: 0.9867, Val Acc: 0.8322
2026-07-08 01:46:43,401 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1980, Val Loss: 0.6201, Train Acc: 0.9658, Val Acc: 0.6510
2026-07-08 01:46:57,767 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1690, Val Loss: 0.5945, Train Acc: 0.9848, Val Acc: 0.6376
2026-07-08 01:47:12,136 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1684, Val Loss: 0.5891, Train Acc: 0.9772, Val Acc: 0.6510
2026-07-08 01:47:26,499 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1612, Val Loss: 0.4837, Train Acc: 0.9848, Val Acc: 0.8255
2026-07-08 01:47:40,874 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1559, Val Loss: 0.6789, Train Acc: 0.9791, Val Acc: 0.5906
2026-07-08 01:47:40,995 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:47:55,369 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1476, Val Loss: 0.5580, Train Acc: 0.9753, Val Acc: 0.8188
2026-07-08 01:48:09,757 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1291, Val Loss: 0.5540, Train Acc: 0.9886, Val Acc: 0.6644
2026-07-08 01:48:24,127 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1334, Val Loss: 0.6869, Train Acc: 0.9753, Val Acc: 0.5772
2026-07-08 01:48:38,499 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1192, Val Loss: 0.6005, Train Acc: 0.9867, Val Acc: 0.5772
2026-07-08 01:48:52,873 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1198, Val Loss: 0.4907, Train Acc: 0.9886, Val Acc: 0.8389
2026-07-08 01:49:07,245 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1051, Val Loss: 0.4545, Train Acc: 0.9829, Val Acc: 0.8322
2026-07-08 01:49:07,358 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:49:21,719 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1160, Val Loss: 0.6059, Train Acc: 0.9715, Val Acc: 0.6174
2026-07-08 01:49:22,408 - INFO - [Train Regular Models] Run 8/30 - Test Loss: 0.4985, Test Accuracy: 0.7423
2026-07-08 01:49:22,485 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_8.pt
2026-07-08 01:49:22,758 - INFO - [Train Regular Models] Training model 9/30...
2026-07-08 01:49:37,230 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5329, Val Loss: 0.7269, Train Acc: 0.8308, Val Acc: 0.4631
2026-07-08 01:49:51,618 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4019, Val Loss: 0.5294, Train Acc: 0.9297, Val Acc: 0.7852
2026-07-08 01:50:06,002 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3455, Val Loss: 0.5451, Train Acc: 0.9601, Val Acc: 0.7114
2026-07-08 01:50:20,394 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3149, Val Loss: 0.5678, Train Acc: 0.9544, Val Acc: 0.7919
2026-07-08 01:50:34,769 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.2864, Val Loss: 0.6303, Train Acc: 0.9810, Val Acc: 0.5839
2026-07-08 01:50:49,167 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2707, Val Loss: 0.6244, Train Acc: 0.9715, Val Acc: 0.6040
2026-07-08 01:50:49,281 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:51:03,677 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2749, Val Loss: 0.5617, Train Acc: 0.9696, Val Acc: 0.6846
2026-07-08 01:51:18,072 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2449, Val Loss: 0.6009, Train Acc: 0.9753, Val Acc: 0.5973
2026-07-08 01:51:32,459 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2096, Val Loss: 0.5512, Train Acc: 0.9829, Val Acc: 0.7114
2026-07-08 01:51:46,841 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2185, Val Loss: 0.6190, Train Acc: 0.9791, Val Acc: 0.5973
2026-07-08 01:52:01,217 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.1917, Val Loss: 0.5245, Train Acc: 0.9810, Val Acc: 0.7919
2026-07-08 01:52:15,583 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1747, Val Loss: 0.5399, Train Acc: 0.9810, Val Acc: 0.7315
2026-07-08 01:52:15,706 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:52:30,084 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1788, Val Loss: 0.5483, Train Acc: 0.9753, Val Acc: 0.7987
2026-07-08 01:52:44,469 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1635, Val Loss: 0.5649, Train Acc: 0.9772, Val Acc: 0.7584
2026-07-08 01:52:58,848 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1428, Val Loss: 0.8318, Train Acc: 0.9829, Val Acc: 0.5436
2026-07-08 01:53:13,219 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1439, Val Loss: 0.6657, Train Acc: 0.9829, Val Acc: 0.5570
2026-07-08 01:53:27,620 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1382, Val Loss: 0.4953, Train Acc: 0.9810, Val Acc: 0.8121
2026-07-08 01:53:42,018 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1411, Val Loss: 0.6155, Train Acc: 0.9753, Val Acc: 0.6040
2026-07-08 01:53:42,135 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:53:56,517 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1370, Val Loss: 0.5925, Train Acc: 0.9810, Val Acc: 0.6711
2026-07-08 01:54:10,888 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1281, Val Loss: 0.5702, Train Acc: 0.9848, Val Acc: 0.6779
2026-07-08 01:54:25,285 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1190, Val Loss: 0.6020, Train Acc: 0.9734, Val Acc: 0.6040
2026-07-08 01:54:39,674 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1165, Val Loss: 0.4796, Train Acc: 0.9772, Val Acc: 0.6980
2026-07-08 01:54:54,068 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1007, Val Loss: 0.4776, Train Acc: 0.9810, Val Acc: 0.8322
2026-07-08 01:55:08,456 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.0959, Val Loss: 0.4534, Train Acc: 0.9810, Val Acc: 0.8322
2026-07-08 01:55:08,575 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 01:55:22,945 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1018, Val Loss: 0.5017, Train Acc: 0.9848, Val Acc: 0.6980
2026-07-08 01:55:23,625 - INFO - [Train Regular Models] Run 9/30 - Test Loss: 0.4081, Test Accuracy: 0.8144
2026-07-08 01:55:23,702 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_9.pt
2026-07-08 01:55:23,978 - INFO - [Train Regular Models] Training model 10/30...
2026-07-08 01:55:38,449 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5502, Val Loss: 0.6700, Train Acc: 0.7490, Val Acc: 0.5235
2026-07-08 01:55:52,819 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4400, Val Loss: 0.7074, Train Acc: 0.8992, Val Acc: 0.4631
2026-07-08 01:56:07,194 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3835, Val Loss: 0.6197, Train Acc: 0.9373, Val Acc: 0.6174
2026-07-08 01:56:21,549 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3455, Val Loss: 0.6379, Train Acc: 0.9715, Val Acc: 0.5503
2026-07-08 01:56:35,919 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3286, Val Loss: 0.5948, Train Acc: 0.9658, Val Acc: 0.6846
2026-07-08 01:56:50,304 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2933, Val Loss: 0.6273, Train Acc: 0.9772, Val Acc: 0.6510
2026-07-08 01:56:50,420 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 01:57:04,800 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2865, Val Loss: 0.5986, Train Acc: 0.9791, Val Acc: 0.6174
2026-07-08 01:57:19,192 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2655, Val Loss: 0.6682, Train Acc: 0.9791, Val Acc: 0.5705
2026-07-08 01:57:33,582 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2447, Val Loss: 0.5758, Train Acc: 0.9772, Val Acc: 0.6376
2026-07-08 01:57:47,970 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2452, Val Loss: 0.6111, Train Acc: 0.9829, Val Acc: 0.6376
2026-07-08 01:58:02,361 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2237, Val Loss: 0.7936, Train Acc: 0.9848, Val Acc: 0.4966
2026-07-08 01:58:16,745 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2220, Val Loss: 0.8597, Train Acc: 0.9810, Val Acc: 0.4631
2026-07-08 01:58:16,862 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 01:58:31,256 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.2072, Val Loss: 0.6203, Train Acc: 0.9829, Val Acc: 0.6309
2026-07-08 01:58:45,650 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1970, Val Loss: 0.6293, Train Acc: 0.9867, Val Acc: 0.5705
2026-07-08 01:59:00,019 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1790, Val Loss: 0.6487, Train Acc: 0.9829, Val Acc: 0.5570
2026-07-08 01:59:14,511 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1648, Val Loss: 0.8208, Train Acc: 0.9886, Val Acc: 0.5168
2026-07-08 01:59:28,883 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1622, Val Loss: 0.6835, Train Acc: 0.9867, Val Acc: 0.5638
2026-07-08 01:59:43,273 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1586, Val Loss: 0.5590, Train Acc: 0.9829, Val Acc: 0.6510
2026-07-08 01:59:43,392 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 01:59:57,803 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1375, Val Loss: 0.6001, Train Acc: 0.9867, Val Acc: 0.6107
2026-07-08 02:00:12,218 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1318, Val Loss: 0.5314, Train Acc: 0.9829, Val Acc: 0.6644
2026-07-08 02:00:26,616 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1271, Val Loss: 0.6448, Train Acc: 0.9867, Val Acc: 0.5906
2026-07-08 02:00:41,013 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1171, Val Loss: 0.8663, Train Acc: 0.9848, Val Acc: 0.5369
2026-07-08 02:00:55,406 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1182, Val Loss: 0.5991, Train Acc: 0.9886, Val Acc: 0.6443
2026-07-08 02:01:09,803 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1148, Val Loss: 0.5206, Train Acc: 0.9810, Val Acc: 0.7450
2026-07-08 02:01:09,917 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:01:24,318 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1065, Val Loss: 0.5463, Train Acc: 0.9848, Val Acc: 0.6711
2026-07-08 02:01:25,009 - INFO - [Train Regular Models] Run 10/30 - Test Loss: 0.4387, Test Accuracy: 0.8041
2026-07-08 02:01:25,086 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_10.pt
2026-07-08 02:01:25,362 - INFO - [Train Regular Models] Training model 11/30...
2026-07-08 02:01:39,825 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5737, Val Loss: 0.6378, Train Acc: 0.7928, Val Acc: 0.6107
2026-07-08 02:01:54,202 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4734, Val Loss: 0.6003, Train Acc: 0.9259, Val Acc: 0.7718
2026-07-08 02:02:08,577 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.4379, Val Loss: 0.5785, Train Acc: 0.9468, Val Acc: 0.7919
2026-07-08 02:02:22,964 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.4043, Val Loss: 0.5678, Train Acc: 0.9696, Val Acc: 0.7785
2026-07-08 02:02:37,333 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3780, Val Loss: 0.6310, Train Acc: 0.9715, Val Acc: 0.5839
2026-07-08 02:02:51,705 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.3578, Val Loss: 0.6243, Train Acc: 0.9791, Val Acc: 0.6242
2026-07-08 02:02:51,821 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 02:03:06,208 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.3328, Val Loss: 0.5416, Train Acc: 0.9772, Val Acc: 0.7987
2026-07-08 02:03:20,586 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.3170, Val Loss: 0.5581, Train Acc: 0.9829, Val Acc: 0.6980
2026-07-08 02:03:34,971 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2934, Val Loss: 0.5505, Train Acc: 0.9848, Val Acc: 0.8054
2026-07-08 02:03:49,350 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2781, Val Loss: 0.6586, Train Acc: 0.9810, Val Acc: 0.5839
2026-07-08 02:04:03,737 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2787, Val Loss: 0.5794, Train Acc: 0.9772, Val Acc: 0.6779
2026-07-08 02:04:18,122 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2526, Val Loss: 0.6403, Train Acc: 0.9772, Val Acc: 0.6980
2026-07-08 02:04:18,242 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 02:04:32,629 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.2444, Val Loss: 0.5919, Train Acc: 0.9867, Val Acc: 0.6644
2026-07-08 02:04:47,011 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.2276, Val Loss: 0.6779, Train Acc: 0.9886, Val Acc: 0.5906
2026-07-08 02:05:01,390 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.2176, Val Loss: 0.4904, Train Acc: 0.9715, Val Acc: 0.8456
2026-07-08 02:05:15,766 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.2005, Val Loss: 0.5043, Train Acc: 0.9810, Val Acc: 0.8389
2026-07-08 02:05:30,140 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1958, Val Loss: 0.5947, Train Acc: 0.9753, Val Acc: 0.6242
2026-07-08 02:05:44,535 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1881, Val Loss: 0.6288, Train Acc: 0.9829, Val Acc: 0.5906
2026-07-08 02:05:44,651 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 02:05:59,047 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1806, Val Loss: 0.5677, Train Acc: 0.9753, Val Acc: 0.6711
2026-07-08 02:06:13,435 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1642, Val Loss: 0.6630, Train Acc: 0.9791, Val Acc: 0.5570
2026-07-08 02:06:27,825 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1576, Val Loss: 0.6012, Train Acc: 0.9848, Val Acc: 0.6242
2026-07-08 02:06:42,201 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1460, Val Loss: 0.7568, Train Acc: 0.9848, Val Acc: 0.5570
2026-07-08 02:06:56,589 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1413, Val Loss: 0.5237, Train Acc: 0.9734, Val Acc: 0.8188
2026-07-08 02:07:10,979 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1429, Val Loss: 0.5940, Train Acc: 0.9848, Val Acc: 0.6309
2026-07-08 02:07:11,088 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:07:25,473 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1317, Val Loss: 0.5790, Train Acc: 0.9829, Val Acc: 0.6376
2026-07-08 02:07:26,160 - INFO - [Train Regular Models] Run 11/30 - Test Loss: 0.4684, Test Accuracy: 0.7629
2026-07-08 02:07:26,237 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_11.pt
2026-07-08 02:07:26,509 - INFO - [Train Regular Models] Training model 12/30...
2026-07-08 02:07:40,976 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5014, Val Loss: 0.6467, Train Acc: 0.8270, Val Acc: 0.5034
2026-07-08 02:07:55,357 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.3659, Val Loss: 0.7217, Train Acc: 0.9373, Val Acc: 0.4631
2026-07-08 02:08:09,756 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3472, Val Loss: 0.6600, Train Acc: 0.9582, Val Acc: 0.5235
2026-07-08 02:08:24,151 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3149, Val Loss: 0.6085, Train Acc: 0.9601, Val Acc: 0.7248
2026-07-08 02:08:38,535 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.2846, Val Loss: 0.8704, Train Acc: 0.9696, Val Acc: 0.4631
2026-07-08 02:08:52,929 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2515, Val Loss: 0.6184, Train Acc: 0.9658, Val Acc: 0.7987
2026-07-08 02:08:53,043 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 02:09:07,423 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2451, Val Loss: 0.8033, Train Acc: 0.9791, Val Acc: 0.4832
2026-07-08 02:09:21,818 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2329, Val Loss: 0.8386, Train Acc: 0.9772, Val Acc: 0.4765
2026-07-08 02:09:36,191 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.1981, Val Loss: 0.7645, Train Acc: 0.9829, Val Acc: 0.5436
2026-07-08 02:09:50,568 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.1819, Val Loss: 0.6328, Train Acc: 0.9829, Val Acc: 0.5705
2026-07-08 02:10:04,952 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.1767, Val Loss: 0.6327, Train Acc: 0.9772, Val Acc: 0.5906
2026-07-08 02:10:19,351 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1699, Val Loss: 0.5347, Train Acc: 0.9810, Val Acc: 0.7047
2026-07-08 02:10:19,467 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 02:10:33,852 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1547, Val Loss: 0.6229, Train Acc: 0.9848, Val Acc: 0.5839
2026-07-08 02:10:48,235 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1524, Val Loss: 0.5005, Train Acc: 0.9696, Val Acc: 0.8054
2026-07-08 02:11:02,620 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1383, Val Loss: 0.4639, Train Acc: 0.9810, Val Acc: 0.8322
2026-07-08 02:11:17,012 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1307, Val Loss: 0.7395, Train Acc: 0.9810, Val Acc: 0.5436
2026-07-08 02:11:31,504 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1213, Val Loss: 0.5418, Train Acc: 0.9848, Val Acc: 0.6913
2026-07-08 02:11:45,883 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1179, Val Loss: 0.6054, Train Acc: 0.9810, Val Acc: 0.5973
2026-07-08 02:11:45,998 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 02:12:00,386 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1218, Val Loss: 0.4810, Train Acc: 0.9867, Val Acc: 0.8792
2026-07-08 02:12:14,783 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1129, Val Loss: 0.5387, Train Acc: 0.9867, Val Acc: 0.6913
2026-07-08 02:12:29,168 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.0999, Val Loss: 0.4775, Train Acc: 0.9848, Val Acc: 0.8322
2026-07-08 02:12:43,570 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.0953, Val Loss: 0.6714, Train Acc: 0.9867, Val Acc: 0.5906
2026-07-08 02:12:57,960 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1186, Val Loss: 0.6726, Train Acc: 0.9753, Val Acc: 0.5839
2026-07-08 02:13:12,355 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.0831, Val Loss: 0.5356, Train Acc: 0.9886, Val Acc: 0.6980
2026-07-08 02:13:12,470 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:13:26,871 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.0916, Val Loss: 0.4519, Train Acc: 0.9829, Val Acc: 0.8255
2026-07-08 02:13:27,557 - INFO - [Train Regular Models] Run 12/30 - Test Loss: 0.3736, Test Accuracy: 0.8866
2026-07-08 02:13:27,630 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_12.pt
2026-07-08 02:13:27,908 - INFO - [Train Regular Models] Training model 13/30...
2026-07-08 02:13:42,386 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5373, Val Loss: 0.6853, Train Acc: 0.8004, Val Acc: 0.4832
2026-07-08 02:13:56,785 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4137, Val Loss: 0.6066, Train Acc: 0.9163, Val Acc: 0.7517
2026-07-08 02:14:11,185 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3811, Val Loss: 0.5632, Train Acc: 0.9449, Val Acc: 0.7651
2026-07-08 02:14:25,593 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3382, Val Loss: 0.6917, Train Acc: 0.9639, Val Acc: 0.4631
2026-07-08 02:14:39,990 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3340, Val Loss: 0.6067, Train Acc: 0.9658, Val Acc: 0.7651
2026-07-08 02:14:54,386 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2977, Val Loss: 0.6293, Train Acc: 0.9867, Val Acc: 0.5436
2026-07-08 02:14:54,506 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 02:15:08,906 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2713, Val Loss: 0.5588, Train Acc: 0.9810, Val Acc: 0.8054
2026-07-08 02:15:23,307 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2502, Val Loss: 0.5615, Train Acc: 0.9829, Val Acc: 0.7047
2026-07-08 02:15:37,705 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2361, Val Loss: 0.5816, Train Acc: 0.9848, Val Acc: 0.7114
2026-07-08 02:15:52,101 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2254, Val Loss: 0.6413, Train Acc: 0.9772, Val Acc: 0.5839
2026-07-08 02:16:06,486 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2263, Val Loss: 0.7806, Train Acc: 0.9829, Val Acc: 0.4765
2026-07-08 02:16:20,875 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2023, Val Loss: 0.6281, Train Acc: 0.9810, Val Acc: 0.5638
2026-07-08 02:16:20,992 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 02:16:35,373 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.2004, Val Loss: 0.5495, Train Acc: 0.9810, Val Acc: 0.7919
2026-07-08 02:16:49,769 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1863, Val Loss: 0.4924, Train Acc: 0.9867, Val Acc: 0.8389
2026-07-08 02:17:04,153 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1892, Val Loss: 0.5714, Train Acc: 0.9772, Val Acc: 0.7114
2026-07-08 02:17:18,551 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1812, Val Loss: 0.6053, Train Acc: 0.9772, Val Acc: 0.7383
2026-07-08 02:17:32,952 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1661, Val Loss: 0.5767, Train Acc: 0.9772, Val Acc: 0.7315
2026-07-08 02:17:47,352 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1551, Val Loss: 0.6524, Train Acc: 0.9886, Val Acc: 0.5570
2026-07-08 02:17:47,466 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 02:18:01,854 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1572, Val Loss: 0.6584, Train Acc: 0.9810, Val Acc: 0.5570
2026-07-08 02:18:16,250 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1397, Val Loss: 0.5408, Train Acc: 0.9867, Val Acc: 0.6846
2026-07-08 02:18:30,655 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1253, Val Loss: 0.4883, Train Acc: 0.9829, Val Acc: 0.7919
2026-07-08 02:18:45,069 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1345, Val Loss: 0.4532, Train Acc: 0.9829, Val Acc: 0.8054
2026-07-08 02:18:59,467 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1168, Val Loss: 0.5998, Train Acc: 0.9867, Val Acc: 0.5906
2026-07-08 02:19:13,853 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1178, Val Loss: 0.7517, Train Acc: 0.9829, Val Acc: 0.5369
2026-07-08 02:19:13,966 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:19:28,368 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1227, Val Loss: 0.5642, Train Acc: 0.9791, Val Acc: 0.6644
2026-07-08 02:19:29,057 - INFO - [Train Regular Models] Run 13/30 - Test Loss: 0.4913, Test Accuracy: 0.8041
2026-07-08 02:19:29,131 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_13.pt
2026-07-08 02:19:29,411 - INFO - [Train Regular Models] Training model 14/30...
2026-07-08 02:19:43,884 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5824, Val Loss: 0.6753, Train Acc: 0.7586, Val Acc: 0.4899
2026-07-08 02:19:58,263 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4201, Val Loss: 0.6247, Train Acc: 0.9316, Val Acc: 0.6376
2026-07-08 02:20:12,649 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.3671, Val Loss: 0.6125, Train Acc: 0.9392, Val Acc: 0.5503
2026-07-08 02:20:27,019 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3214, Val Loss: 0.5799, Train Acc: 0.9620, Val Acc: 0.6510
2026-07-08 02:20:41,421 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3110, Val Loss: 0.7036, Train Acc: 0.9582, Val Acc: 0.4966
2026-07-08 02:20:55,817 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.2787, Val Loss: 0.6757, Train Acc: 0.9791, Val Acc: 0.5168
2026-07-08 02:20:55,933 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 02:21:10,324 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.2543, Val Loss: 0.7379, Train Acc: 0.9810, Val Acc: 0.4698
2026-07-08 02:21:24,717 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2481, Val Loss: 0.6095, Train Acc: 0.9772, Val Acc: 0.5906
2026-07-08 02:21:39,104 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2215, Val Loss: 0.6555, Train Acc: 0.9810, Val Acc: 0.5570
2026-07-08 02:21:53,500 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2087, Val Loss: 0.6492, Train Acc: 0.9829, Val Acc: 0.5705
2026-07-08 02:22:07,888 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.1864, Val Loss: 0.6208, Train Acc: 0.9848, Val Acc: 0.6309
2026-07-08 02:22:22,284 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.1882, Val Loss: 0.5751, Train Acc: 0.9715, Val Acc: 0.6510
2026-07-08 02:22:22,400 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 02:22:36,801 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.1818, Val Loss: 0.5825, Train Acc: 0.9810, Val Acc: 0.6510
2026-07-08 02:22:51,205 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.1799, Val Loss: 0.6354, Train Acc: 0.9772, Val Acc: 0.6242
2026-07-08 02:23:05,609 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.1668, Val Loss: 0.5989, Train Acc: 0.9734, Val Acc: 0.7517
2026-07-08 02:23:20,005 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.1723, Val Loss: 0.5868, Train Acc: 0.9810, Val Acc: 0.6242
2026-07-08 02:23:34,498 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1485, Val Loss: 0.7134, Train Acc: 0.9829, Val Acc: 0.5436
2026-07-08 02:23:48,917 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1461, Val Loss: 0.4996, Train Acc: 0.9753, Val Acc: 0.8255
2026-07-08 02:23:49,033 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 02:24:03,438 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1299, Val Loss: 0.6209, Train Acc: 0.9886, Val Acc: 0.5839
2026-07-08 02:24:17,824 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1235, Val Loss: 0.4278, Train Acc: 0.9791, Val Acc: 0.8591
2026-07-08 02:24:32,218 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1159, Val Loss: 0.4588, Train Acc: 0.9848, Val Acc: 0.8322
2026-07-08 02:24:46,611 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1062, Val Loss: 0.4767, Train Acc: 0.9810, Val Acc: 0.8255
2026-07-08 02:25:01,002 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.0997, Val Loss: 0.5401, Train Acc: 0.9886, Val Acc: 0.6510
2026-07-08 02:25:15,411 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1179, Val Loss: 0.5590, Train Acc: 0.9791, Val Acc: 0.6443
2026-07-08 02:25:15,524 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:25:29,908 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.0982, Val Loss: 0.5710, Train Acc: 0.9848, Val Acc: 0.6779
2026-07-08 02:25:30,591 - INFO - [Train Regular Models] Run 14/30 - Test Loss: 0.5050, Test Accuracy: 0.8041
2026-07-08 02:25:30,665 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_14.pt
2026-07-08 02:25:30,947 - INFO - [Train Regular Models] Training model 15/30...
2026-07-08 02:25:45,417 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5497, Val Loss: 0.6083, Train Acc: 0.7605, Val Acc: 0.7651
2026-07-08 02:25:59,801 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4380, Val Loss: 0.8154, Train Acc: 0.9392, Val Acc: 0.4765
2026-07-08 02:26:14,187 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.4021, Val Loss: 0.6158, Train Acc: 0.9449, Val Acc: 0.5839
2026-07-08 02:26:28,595 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.3787, Val Loss: 0.6743, Train Acc: 0.9582, Val Acc: 0.5101
2026-07-08 02:26:42,982 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3443, Val Loss: 0.7306, Train Acc: 0.9696, Val Acc: 0.5235
2026-07-08 02:26:57,372 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.3150, Val Loss: 0.5589, Train Acc: 0.9791, Val Acc: 0.6980
2026-07-08 02:26:57,491 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 02:27:11,874 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.3035, Val Loss: 0.6195, Train Acc: 0.9677, Val Acc: 0.6443
2026-07-08 02:27:26,254 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.2753, Val Loss: 0.5477, Train Acc: 0.9810, Val Acc: 0.6711
2026-07-08 02:27:40,649 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.2682, Val Loss: 0.5580, Train Acc: 0.9715, Val Acc: 0.6443
2026-07-08 02:27:55,035 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.2444, Val Loss: 0.5734, Train Acc: 0.9791, Val Acc: 0.6443
2026-07-08 02:28:09,435 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2453, Val Loss: 0.5610, Train Acc: 0.9639, Val Acc: 0.7919
2026-07-08 02:28:23,831 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2206, Val Loss: 0.5219, Train Acc: 0.9867, Val Acc: 0.6577
2026-07-08 02:28:23,948 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 02:28:38,333 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.2058, Val Loss: 1.0627, Train Acc: 0.9772, Val Acc: 0.4966
2026-07-08 02:28:52,701 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.2075, Val Loss: 0.5383, Train Acc: 0.9829, Val Acc: 0.7114
2026-07-08 02:29:07,092 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.2074, Val Loss: 0.6968, Train Acc: 0.9715, Val Acc: 0.5436
2026-07-08 02:29:21,469 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.2040, Val Loss: 0.5975, Train Acc: 0.9658, Val Acc: 0.6644
2026-07-08 02:29:35,851 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.1877, Val Loss: 0.6027, Train Acc: 0.9753, Val Acc: 0.7047
2026-07-08 02:29:50,253 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1837, Val Loss: 0.9260, Train Acc: 0.9753, Val Acc: 0.4966
2026-07-08 02:29:50,367 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 02:30:04,754 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.1562, Val Loss: 0.7276, Train Acc: 0.9810, Val Acc: 0.5369
2026-07-08 02:30:19,157 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1532, Val Loss: 0.4938, Train Acc: 0.9810, Val Acc: 0.8389
2026-07-08 02:30:33,542 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1471, Val Loss: 0.7677, Train Acc: 0.9810, Val Acc: 0.5436
2026-07-08 02:30:47,928 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1459, Val Loss: 0.5852, Train Acc: 0.9772, Val Acc: 0.6040
2026-07-08 02:31:02,311 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1180, Val Loss: 0.5635, Train Acc: 0.9810, Val Acc: 0.6376
2026-07-08 02:31:16,700 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1194, Val Loss: 0.5958, Train Acc: 0.9848, Val Acc: 0.5906
2026-07-08 02:31:16,814 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:31:31,203 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1238, Val Loss: 0.5518, Train Acc: 0.9829, Val Acc: 0.6309
2026-07-08 02:31:31,890 - INFO - [Train Regular Models] Run 15/30 - Test Loss: 0.4224, Test Accuracy: 0.7526
2026-07-08 02:31:31,963 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_15.pt
2026-07-08 02:31:32,246 - INFO - [Train Regular Models] Training model 16/30...
2026-07-08 02:31:46,711 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5793, Val Loss: 0.6642, Train Acc: 0.7890, Val Acc: 0.4899
2026-07-08 02:32:01,104 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4929, Val Loss: 0.6100, Train Acc: 0.9144, Val Acc: 0.7718
2026-07-08 02:32:15,501 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.4490, Val Loss: 0.5795, Train Acc: 0.9563, Val Acc: 0.7718
2026-07-08 02:32:29,889 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.4169, Val Loss: 0.5714, Train Acc: 0.9620, Val Acc: 0.8456
2026-07-08 02:32:44,289 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.3938, Val Loss: 0.5628, Train Acc: 0.9639, Val Acc: 0.7248
2026-07-08 02:32:58,670 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.3682, Val Loss: 0.5880, Train Acc: 0.9829, Val Acc: 0.6913
2026-07-08 02:32:58,787 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 02:33:13,196 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.3465, Val Loss: 0.5732, Train Acc: 0.9829, Val Acc: 0.7114
2026-07-08 02:33:27,582 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.3454, Val Loss: 0.6779, Train Acc: 0.9677, Val Acc: 0.6040
2026-07-08 02:33:41,986 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.3157, Val Loss: 0.5785, Train Acc: 0.9829, Val Acc: 0.7785
2026-07-08 02:33:56,386 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.3129, Val Loss: 0.7158, Train Acc: 0.9772, Val Acc: 0.5302
2026-07-08 02:34:10,778 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.3033, Val Loss: 0.6190, Train Acc: 0.9810, Val Acc: 0.6242
2026-07-08 02:34:25,170 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2732, Val Loss: 0.6267, Train Acc: 0.9734, Val Acc: 0.6107
2026-07-08 02:34:25,291 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 02:34:39,692 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.2628, Val Loss: 0.6787, Train Acc: 0.9772, Val Acc: 0.5638
2026-07-08 02:34:54,084 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.2475, Val Loss: 0.6554, Train Acc: 0.9829, Val Acc: 0.5772
2026-07-08 02:35:08,484 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.2280, Val Loss: 0.5506, Train Acc: 0.9810, Val Acc: 0.6779
2026-07-08 02:35:22,891 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.2239, Val Loss: 0.5756, Train Acc: 0.9791, Val Acc: 0.6779
2026-07-08 02:35:37,280 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.2134, Val Loss: 0.6482, Train Acc: 0.9848, Val Acc: 0.5839
2026-07-08 02:35:51,671 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.1995, Val Loss: 0.5583, Train Acc: 0.9772, Val Acc: 0.7315
2026-07-08 02:35:51,790 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 02:36:06,178 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.2016, Val Loss: 0.5753, Train Acc: 0.9772, Val Acc: 0.6174
2026-07-08 02:36:20,583 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1858, Val Loss: 0.5790, Train Acc: 0.9753, Val Acc: 0.7114
2026-07-08 02:36:34,976 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1644, Val Loss: 0.5443, Train Acc: 0.9848, Val Acc: 0.7517
2026-07-08 02:36:49,368 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1609, Val Loss: 0.8919, Train Acc: 0.9810, Val Acc: 0.5369
2026-07-08 02:37:03,773 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1586, Val Loss: 0.5831, Train Acc: 0.9829, Val Acc: 0.7517
2026-07-08 02:37:18,169 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1719, Val Loss: 0.6139, Train Acc: 0.9772, Val Acc: 0.6107
2026-07-08 02:37:18,286 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:37:32,684 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1457, Val Loss: 0.6443, Train Acc: 0.9848, Val Acc: 0.6376
2026-07-08 02:37:33,385 - INFO - [Train Regular Models] Run 16/30 - Test Loss: 0.5474, Test Accuracy: 0.8144
2026-07-08 02:37:33,459 - INFO - [Train Regular Models] Model saved to outputs/exp1/normal_models/model_16.pt
2026-07-08 02:37:33,741 - INFO - [Train Regular Models] Training model 17/30...
2026-07-08 02:37:48,208 - INFO - [Train Regular Models] Epoch [1/25], Train Loss: 0.5526, Val Loss: 0.7408, Train Acc: 0.7243, Val Acc: 0.4631
2026-07-08 02:38:02,589 - INFO - [Train Regular Models] Epoch [2/25], Train Loss: 0.4890, Val Loss: 0.6944, Train Acc: 0.7947, Val Acc: 0.5034
2026-07-08 02:38:16,985 - INFO - [Train Regular Models] Epoch [3/25], Train Loss: 0.4453, Val Loss: 0.6712, Train Acc: 0.8593, Val Acc: 0.5570
2026-07-08 02:38:31,384 - INFO - [Train Regular Models] Epoch [4/25], Train Loss: 0.4263, Val Loss: 0.7043, Train Acc: 0.8802, Val Acc: 0.5436
2026-07-08 02:38:45,757 - INFO - [Train Regular Models] Epoch [5/25], Train Loss: 0.4018, Val Loss: 0.7160, Train Acc: 0.9259, Val Acc: 0.5369
2026-07-08 02:39:00,141 - INFO - [Train Regular Models] Epoch [6/25], Train Loss: 0.3768, Val Loss: 0.7263, Train Acc: 0.9468, Val Acc: 0.5235
2026-07-08 02:39:00,262 - INFO - [Train Regular Models] Model saved at epoch 6
2026-07-08 02:39:14,652 - INFO - [Train Regular Models] Epoch [7/25], Train Loss: 0.3657, Val Loss: 0.6476, Train Acc: 0.9544, Val Acc: 0.6443
2026-07-08 02:39:29,041 - INFO - [Train Regular Models] Epoch [8/25], Train Loss: 0.3527, Val Loss: 0.6929, Train Acc: 0.9639, Val Acc: 0.5503
2026-07-08 02:39:43,407 - INFO - [Train Regular Models] Epoch [9/25], Train Loss: 0.3287, Val Loss: 0.6200, Train Acc: 0.9772, Val Acc: 0.6174
2026-07-08 02:39:57,800 - INFO - [Train Regular Models] Epoch [10/25], Train Loss: 0.3184, Val Loss: 0.6237, Train Acc: 0.9772, Val Acc: 0.5705
2026-07-08 02:40:12,186 - INFO - [Train Regular Models] Epoch [11/25], Train Loss: 0.2984, Val Loss: 0.6274, Train Acc: 0.9791, Val Acc: 0.5906
2026-07-08 02:40:26,582 - INFO - [Train Regular Models] Epoch [12/25], Train Loss: 0.2736, Val Loss: 0.5997, Train Acc: 0.9867, Val Acc: 0.6040
2026-07-08 02:40:26,709 - INFO - [Train Regular Models] Model saved at epoch 12
2026-07-08 02:40:41,101 - INFO - [Train Regular Models] Epoch [13/25], Train Loss: 0.2771, Val Loss: 0.9275, Train Acc: 0.9772, Val Acc: 0.5034
2026-07-08 02:40:55,485 - INFO - [Train Regular Models] Epoch [14/25], Train Loss: 0.2693, Val Loss: 0.6550, Train Acc: 0.9791, Val Acc: 0.5705
2026-07-08 02:41:09,889 - INFO - [Train Regular Models] Epoch [15/25], Train Loss: 0.2401, Val Loss: 0.5919, Train Acc: 0.9829, Val Acc: 0.6174
2026-07-08 02:41:24,286 - INFO - [Train Regular Models] Epoch [16/25], Train Loss: 0.2297, Val Loss: 0.5206, Train Acc: 0.9772, Val Acc: 0.8322
2026-07-08 02:41:38,683 - INFO - [Train Regular Models] Epoch [17/25], Train Loss: 0.2200, Val Loss: 0.6000, Train Acc: 0.9848, Val Acc: 0.5973
2026-07-08 02:41:53,056 - INFO - [Train Regular Models] Epoch [18/25], Train Loss: 0.2094, Val Loss: 0.6548, Train Acc: 0.9829, Val Acc: 0.5705
2026-07-08 02:41:53,176 - INFO - [Train Regular Models] Model saved at epoch 18
2026-07-08 02:42:07,573 - INFO - [Train Regular Models] Epoch [19/25], Train Loss: 0.2113, Val Loss: 0.4974, Train Acc: 0.9696, Val Acc: 0.8054
2026-07-08 02:42:21,975 - INFO - [Train Regular Models] Epoch [20/25], Train Loss: 0.1932, Val Loss: 0.6701, Train Acc: 0.9867, Val Acc: 0.5705
2026-07-08 02:42:36,356 - INFO - [Train Regular Models] Epoch [21/25], Train Loss: 0.1937, Val Loss: 0.6473, Train Acc: 0.9791, Val Acc: 0.5772
2026-07-08 02:42:50,741 - INFO - [Train Regular Models] Epoch [22/25], Train Loss: 0.1737, Val Loss: 0.6145, Train Acc: 0.9867, Val Acc: 0.5772
2026-07-08 02:43:05,117 - INFO - [Train Regular Models] Epoch [23/25], Train Loss: 0.1573, Val Loss: 0.5361, Train Acc: 0.9810, Val Acc: 0.7517
2026-07-08 02:43:19,512 - INFO - [Train Regular Models] Epoch [24/25], Train Loss: 0.1608, Val Loss: 0.4893, Train Acc: 0.9791, Val Acc: 0.8054
2026-07-08 02:43:19,625 - INFO - [Train Regular Models] Model saved at epoch 24
2026-07-08 02:43:34,006 - INFO - [Train Regular Models] Epoch [25/25], Train Loss: 0.1539, Val Loss: 0.5199, Train Acc: 0.9791, Val Acc: 0.7919
2026-07-08 02:43:34,287 - ERROR - Pipeline failed: CUDA out of memory. Tried to allocate 46.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 18.62 MiB is free. Including non-PyTorch memory, this process has 9.81 GiB memory in use. Of the allocated memory 9.27 GiB is allocated by PyTorch, and 230.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Traceback (most recent call last):
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/main.py", line 256, in run_background_pipeline
    task_func(task_tracker, task_logger, config, pipeline_state)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/tasks/train_normal.py", line 105, in train_normal_task
    test_loss, test_acc = tn.test_model(
                          ~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<5 lines>...
        pause_event=pause_event,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/training.py", line 108, in test_model
    outputs = model((mri, xls))
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 66, in forward
    image_out = self.image_section(image_data)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/cnn.py", line 37, in forward
    x = self.splitcnv(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 68, in forward
    right = self.rightblock(right)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/container.py", line 253, in forward
    input = module(input)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 169, in forward
    x = self.cnv(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/model/layers.py", line 30, in forward
    x = self.depthwise(x)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/conv.py", line 735, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nschense/Documents/Medphys_Research/alnn_rewrite/.venv/lib64/python3.14/site-packages/torch/nn/modules/conv.py", line 730, in _conv_forward
    return F.conv3d(
           ~~~~~~~~^
        input, weight, bias, self.stride, self.padding, self.dilation, self.groups
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 46.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 18.62 MiB is free. Including non-PyTorch memory, this process has 9.81 GiB memory in use. Of the allocated memory 9.27 GiB is allocated by PyTorch, and 230.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
2026-07-08 02:43:34,289 - ERROR - [SYSTEM] ERROR: CUDA out of memory. Tried to allocate 46.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 18.62 MiB is free. Including non-PyTorch memory, this process has 9.81 GiB memory in use. Of the allocated memory 9.27 GiB is allocated by PyTorch, and 230.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)